Hi Chris, I understand your point, sounds good to me. So in short, we should opt for an internal-only API, and preferably a single server solution. Is that right?
Thanks Daniel Chris Egerton <chr...@aiven.io.invalid> ezt írta (időpont: 2022. aug. 26., P, 17:36): > Hi Daniel, > > Glad to hear from you! > > With regards to the stripped-down REST API alternative, I don't see how > this would prevent us from introducing the fully-fledged Connect REST API, > or even an augmented variant of it, at some point down the road. If we go > with the internal-only API now, and want to expand later, can't we gate the > expansion behind a feature flag configuration property that by default > disables the new feature? > > I'm also not sure that we'd ever want to expose the raw Connect REST API > for dedicated MM2 clusters. If people want that capability, they can > already spin up a vanilla Connect cluster and run as many MM2 connectors as > they'd like on it, and as of KIP-458 [1], it's even possible to use a > single Connect cluster to replicate between any two Kafka clusters instead > of only targeting the Kafka cluster that the vanilla Connect cluster > operates on top of. I do agree that it'd be great to be able to dynamically > adjust things like topic filters without having to restart a dedicated MM2 > node; I'm just not sure that the vanilla Connect REST API is the > appropriate way to do that, especially since the exact mechanisms that make > a single Connect cluster viable for replicating across any two Kafka > clusters could be abused and cause a dedicated MM2 cluster to start writing > to a completely different Kafka cluster that's not even defined in its > config file. > > Finally, as far as security goes--since this is essentially a bug fix, I'm > inclined to make it as easy as possible for users to adopt it. MTLS is a > great start for securing a REST API, but it's not sufficient on its own > since anyone who could issue an authenticated REST request against the MM2 > cluster would still be able to make any changes they want (with the > exception of accessing internal endpoints, which were secured with > KIP-507). If we were to bring up the fully-fledged Connect REST API, > cluster administrators would also likely have to add some kind of > authorization layer to prevent people from using the REST API to mess with > the configurations of the connectors that MM2 brought up. One way of doing > that is to add a REST extension to your Connect cluster, but implementing > and configuring one in order to be able to run a multi-node MM2 cluster > without hitting this bug seems like too much work to be worth it. > > I think if we had a better picture of what a REST API for dedicated MM2 > clusters would/should look like, then it would be easier to go along with > this, and we could even just add the feature flag in this KIP right now to > address any security concerns. My instinct would be to address this in a > follow-up KIP in order to reduce scope creep, though, and keep this KIP > focused on addressing the bug with multi-node dedicated MM2 clusters. What > do you think? > > Cheers, > > Chris > > [1] - > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy > > On Thu, Aug 25, 2022 at 3:55 AM Dániel Urbán <urb.dani...@gmail.com> > wrote: > > > Hi Chris, > > > > Thanks for bringing this up again :) > > > > 1. I think that is reasonable, though I find the current state of MM2 to > be > > confusing. The current issue with the distributed mode is not documented > > properly, but maybe the logging will help a bit. > > > > 2. Going for an internal-only Connect REST version would lock MM2 out of > a > > path where the REST API can be used to dynamically reconfigure > > replications. For now, I agree, it would be easy to corrupt the state of > > MM2 if someone wanted to use the properties and the REST at the same > time, > > but in the future, we might have a chance to introduce a different config > > mechanism, where only the cluster connections have to be specified in the > > properties file, and everything else can be configured through REST > > (enabling replications, changing topic filters, etc.). Because of this, > I'm > > leaning towards a full Connect REST API. To avoid issues with conflicts > > between the props file and the REST, we could document security best > > practices (e.g. turn on basic auth or mTLS on the Connect REST to avoid > > unwanted interactions). > > > > 3. That is a good point, and I agree, a big plus for motivation. > > > > I have a working version of this in which all flows spin up a dedicated > > Connect REST, but I can give other solutions a try, too. > > > > Thanks, > > Daniel > > > > Chris Egerton <chr...@aiven.io.invalid> ezt írta (időpont: 2022. aug. > 24., > > Sze, 17:46): > > > > > Hi Daniel, > > > > > > I'd like to resurface this KIP in case you're still interested in > > pursuing > > > it. I know it's been a while since you published it, and it hasn't > > received > > > much attention, but I'm hoping we can give it a try now and finally put > > > this long-standing bug to rest. To that end, I have some thoughts about > > the > > > proposal. This isn't a complete review, but I wanted to give enough to > > get > > > the ball rolling: > > > > > > 1. Some environments with firewalls or strict security policies may not > > be > > > able to bring up a REST server for each MM2 node. If we decide that > we'd > > > like to use the Connect REST API (or even just parts of it) to address > > this > > > bug with MM2, it does make sense to eventually make the availability of > > the > > > REST API a hard requirement for running MM2, but it might be a bit too > > > abrupt to do that all in a single release. What do you think about > making > > > the REST API optional for now, but noting that it will become required > > in a > > > later release (probably 4.0.0 or, if that's not enough time, 5.0.0)? We > > > could choose not to bring the REST server for any node whose > > configuration > > > doesn't explicitly opt into one, and maybe log a warning message on > > startup > > > if none is configured. In effect, we'd be marking the current mode (no > > REST > > > server) as deprecated. > > > > > > 2. I'm not sure that we should count out the "Creating an internal-only > > > derivation of the Connect REST API" rejected alternative. Right now, > the > > > single source of truth for the configuration of a MM2 cluster (assuming > > > it's being run in dedicated mode, and not as a connector in a vanilla > > > Connect cluster) is the configuration file used for the process. By > > > bringing up the REST API, we'd expose endpoints to modify connector > > > configurations, which would not only add complexity to the operation > of a > > > MM2 cluster, but even qualify as an attack vector for malicious > entities. > > > Thanks to KIP-507 we have some amount of security around the > > internal-only > > > endpoints used by the Connect framework, but for any public endpoints, > > the > > > Connect REST API doesn't come with any security out of the box. > > > > > > 3. Small point, but with support for exactly-once source connectors > > coming > > > out in 3.3.0, it's also worth noting that that's another feature that > > won't > > > work properly with multi-node MM2 clusters without adding a REST server > > for > > > each node (or some substitute that accomplishes the same goal). I don't > > > think this will affect the direction of the design discussion too much, > > but > > > it does help strengthen the motivation. > > > > > > Cheers, > > > > > > Chris > > > > > > On 2021/02/18 15:57:36 Dániel Urbán wrote: > > > > Hello everyone, > > > > > > > > * Sorry, I meant KIP-710. > > > > > > > > Right now the MirrorMaker cluster is somewhat unreliable, and not > > > > supporting running in a cluster properly. I'd say that fixing this > > would > > > be > > > > a nice addition. > > > > Does anyone have some input on this? > > > > > > > > Thanks in advance > > > > Daniel > > > > > > > > Dániel Urbán <ur...@gmail.com> ezt írta (időpont: 2021. jan. 26., K, > > > > 15:56): > > > > > > > > > Hello everyone, > > > > > > > > > > I would like to start a discussion on KIP-709, which addresses some > > > > > missing features in MM2 dedicated mode. > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters > > > > > > > > > > Currently, the dedicated mode of MM2 does not fully support running > > in > > > a > > > > > cluster. The core issue is that the Connect REST Server is not > > included > > > in > > > > > the dedicated mode, which makes follower->leader communication > > > impossible. > > > > > In some cases, this results in the cluster not being able to react > to > > > > > dynamic configuration changes (e.g. dynamic topic filter changes). > > > > > Another smaller detail is that MM2 dedicated mode eagerly resolves > > > config > > > > > provider references in the Connector configurations, which is > > > undesirable > > > > > and a breaking change compared to vanilla Connect. This can cause > an > > > issue > > > > > for example when there is an environment variable reference, which > > > contains > > > > > some host-specific information, like a file path. The leader > resolves > > > the > > > > > reference eagerly, and the resolved value is propagated to other > MM2 > > > nodes > > > > > instead of the reference being resolved locally, separately on each > > > node. > > > > > > > > > > The KIP addresses these by adding the Connect REST Server to the > MM2 > > > > > dedicated mode for each replication flow, and postponing the config > > > > > provider reference resolution. > > > > > > > > > > Please discuss, I know this is a major change, but also an > important > > > > > feature for MM2 users. > > > > > > > > > > Daniel > > > > > > > > > > > > > > >
