Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2023-01-11 Thread Dániel Urbán
Hi Mickael, Thanks for the input, I updated the KIP with the config name you suggested. Daniel Mickael Maison ezt írta (időpont: 2022. nov. 7., H, 10:48): > Hi Daniel, > > Thanks for the KIP. > > It would be nice to expose more of the REST API as some endpoints are > really useful, for example /

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-11-07 Thread Mickael Maison
Hi Daniel, Thanks for the KIP. It would be nice to expose more of the REST API as some endpoints are really useful, for example /admin or /connectors//tasks-config. However as dedicated mode is currently unusable, I think the approach of "just fixing it" by exposing the internal endpoints is fine

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-09-27 Thread Chris Egerton
Thanks Daniel! No further comments from me, looks good. On Tue, Sep 27, 2022 at 4:39 AM Dániel Urbán wrote: > Hi Chris, > > I understand your points, and I agree that this path is safer in terms of > future plans, I accept it. > Updated the KIP accordingly. > > Thanks, > Daniel > > Chris Egerton

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-09-27 Thread Dániel Urbán
Hi Chris, I understand your points, and I agree that this path is safer in terms of future plans, I accept it. Updated the KIP accordingly. Thanks, Daniel Chris Egerton ezt írta (időpont: 2022. szept. 21., Sze, 18:54): > Hi Daniel, > > I'm a little hesitant to add support for REST extensions a

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-09-21 Thread Chris Egerton
Hi Daniel, I'm a little hesitant to add support for REST extensions and the admin API to dedicated MM2 nodes because they may restrict our options in the future if/when we add a public-facing REST API. Regarding REST extensions specifically, I understand their security value for public-facing API

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-09-21 Thread Dániel Urbán
Hi Chris, About the worker id: makes sense. I changed the wording, but kept it listed as it is a change compared to existing MM2 code. Updated the KIP accordingly. About the REST server configurations: again, I agree, it should be generalized. But I'm not sure about those exceptions you listed, a

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-09-20 Thread Chris Egerton
Hi Daniel, Looking pretty good! Regarding the worker ID generation--apologies, I was unclear with my question. I was wondering if we had to adopt any special logic at all for MM2, or if we could use the same logic that vanilla Kafka Connect does in distributed mode, where the worker ID is its adve

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-09-16 Thread Dániel Urbán
Hi Chris, I went through the KIP and updated it based on our discussion. I think your suggestions simplified (and shortened) the KIP significantly. Thanks, Daniel Dániel Urbán ezt írta (időpont: 2022. szept. 16., P, 15:15): > Hi Chris, > > 1. For the REST-server-per-flow setup, it made sense t

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-09-16 Thread Dániel Urbán
Hi Chris, 1. For the REST-server-per-flow setup, it made sense to introduce some simplified configuration. With a single REST server, it doesn't make sense anymore, I'm removing it from the KIP. 2. I think that changing the worker ID generation still makes sense, otherwise there is no way to diffe

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-08-31 Thread Chris Egerton
Hi Daniel, I've taken a look at the KIP in detail. Here are my complete thoughts (minus the aforementioned sections that may be affected by changes to an internal-only REST API): 1. Why introduce new mm.host.name and mm.rest.protocol properties instead of using the properties that are already use

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-08-29 Thread Chris Egerton
Hi Daniel, Yeah, I think that's the way to go. Adding multiple servers for each herder seems like it'd be too much of a pain for users to configure, and if we keep the API strictly internal for now, we shouldn't be painting ourselves into too much of a corner if/when we decide to expose a public-f

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-08-29 Thread Dániel Urbán
Hi Chris, I understand your point, sounds good to me. So in short, we should opt for an internal-only API, and preferably a single server solution. Is that right? Thanks Daniel Chris Egerton ezt írta (időpont: 2022. aug. 26., P, 17:36): > Hi Daniel, > > Glad to hear from you! > > With regards

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-08-26 Thread Chris Egerton
Hi Daniel, Glad to hear from you! With regards to the stripped-down REST API alternative, I don't see how this would prevent us from introducing the fully-fledged Connect REST API, or even an augmented variant of it, at some point down the road. If we go with the internal-only API now, and want t

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-08-25 Thread Dániel Urbán
Hi Chris, Thanks for bringing this up again :) 1. I think that is reasonable, though I find the current state of MM2 to be confusing. The current issue with the distributed mode is not documented properly, but maybe the logging will help a bit. 2. Going for an internal-only Connect REST version

RE: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

2022-08-24 Thread Chris Egerton
Hi Daniel, I'd like to resurface this KIP in case you're still interested in pursuing it. I know it's been a while since you published it, and it hasn't received much attention, but I'm hoping we can give it a try now and finally put this long-standing bug to rest. To that end, I have some thought