Re: What happened to KIP-158?
Hi Tom. Good timing for your question. I'll be revisiting KIP-158 and I'll bring an updated proposal for discussion this quarter. As always, it'll be ideal to strike a good balance between flexibility and conciseness on what we expose to Connect's developers and operators. Stay tuned. Konstantine On Mon, Nov 25, 2019 at 9:48 AM Ryanne Dolan wrote: > Hey Tom. The approach I ended up taking with MM2 is to construct an > AdminClient based on properties in the connector config, which requires > that such properties exist in two places (the worker config and the > connector config). I still believe that, ideally, a connector implementer > should have access to an admin client based on the worker config, or > perhaps just the properties (so that an admin client can be constructed). > Otherwise, as you point out, there is no way for a connector to control > topics without this workaround. > > Ryanne > > On Mon, Nov 25, 2019, 5:18 AM Tom Bentley wrote: > > > Hi, > > > > KIP-158 proposed a way to allow source connectors a way to configure > their > > topics prior to creation. This was discussed, but the vote didn't get > > enough commiter votes at the time (though I think a couple of the +1s > came > > from people who are now committers). > > > > Meanwhile in KIP-382 (MirrorMaker 2.0) Ryanne made a throwaway comment > > about possibly proposing a KIP which would provide a generic way for > Kafka > > Connect to expose AdminClient functionality to connectors, but I don't > > think that KIP was ever created. > > > > Is there a current plan about how to bring this functionality to > > connectors? If not what's the best way forward, proceed with KIP-158, or > > try for something more generic such as could be used by MM2? > > > > Kind regards, > > > > Tom > > >
Re: What happened to KIP-158?
Hey Tom. The approach I ended up taking with MM2 is to construct an AdminClient based on properties in the connector config, which requires that such properties exist in two places (the worker config and the connector config). I still believe that, ideally, a connector implementer should have access to an admin client based on the worker config, or perhaps just the properties (so that an admin client can be constructed). Otherwise, as you point out, there is no way for a connector to control topics without this workaround. Ryanne On Mon, Nov 25, 2019, 5:18 AM Tom Bentley wrote: > Hi, > > KIP-158 proposed a way to allow source connectors a way to configure their > topics prior to creation. This was discussed, but the vote didn't get > enough commiter votes at the time (though I think a couple of the +1s came > from people who are now committers). > > Meanwhile in KIP-382 (MirrorMaker 2.0) Ryanne made a throwaway comment > about possibly proposing a KIP which would provide a generic way for Kafka > Connect to expose AdminClient functionality to connectors, but I don't > think that KIP was ever created. > > Is there a current plan about how to bring this functionality to > connectors? If not what's the best way forward, proceed with KIP-158, or > try for something more generic such as could be used by MM2? > > Kind regards, > > Tom >
What happened to KIP-158?
Hi, KIP-158 proposed a way to allow source connectors a way to configure their topics prior to creation. This was discussed, but the vote didn't get enough commiter votes at the time (though I think a couple of the +1s came from people who are now committers). Meanwhile in KIP-382 (MirrorMaker 2.0) Ryanne made a throwaway comment about possibly proposing a KIP which would provide a generic way for Kafka Connect to expose AdminClient functionality to connectors, but I don't think that KIP was ever created. Is there a current plan about how to bring this functionality to connectors? If not what's the best way forward, proceed with KIP-158, or try for something more generic such as could be used by MM2? Kind regards, Tom