Re: KIP-769: Connect API to retrieve connector configuration definitions
Hi, I have updated the KIP accordingly. Thanks, Mickael On Fri, Jan 28, 2022 at 7:01 PM Chris Egerton wrote: > > I think a simple approach here is fine. +1 for removing versioning for > other plugins. > > On Fri, Jan 28, 2022 at 11:17 AM Mickael Maison > wrote: > > > Thanks Chris and Tom for the feedback. > > > > I'm having a hard time making a decision on this point! > > > > Re-reading your previous comments, I agree that we're lacking a > > contract on the version field, and we can see value (supporting > > multiple versions for example) if this was improved. > > But I'm starting to wonder if this should be handled in a different KIP. > > > > In this KIP, we can keep the existing Versioned interface for > > Connectors and either not display a version for other plugins or > > hard-code it to something like "N/A". Later if we think we're missing > > the version for other plugins we can decide how to handle it. While it > > would have been nice to harmonize all plugins, I don't think it's a > > requirement or blocker for the rest of the value of this KIP. > > > > Unless you disagree and feel like we need versions for all plugins, > > I'll update the KIP accordingly. > > > > Thanks, > > Mickael > > > > > > > > > > > > On Thu, Jan 27, 2022 at 9:00 PM Tom Bentley wrote: > > > > > > To be honest I'm very sceptical that you can evolve (without breaking > > > compatibility) from a situation of having a large number of plugins which > > > have already implemented version() to return whatever they please, to a > > > situation where you can rely on it as the basis for some feature to > > support > > > multiple versions of plugins. It would be much safer to introduce a new > > > versioning concept, perhaps using Jar manifests and > > > "Specification-Version", than to try to build on Versioned. > > > > > > So I would prefer Javadoc that clearly said that this version number was > > a > > > means for the plugin author to identify the version of their plugin in > > use > > > by users, for the purpose of bug reporting. That at least makes is clear > > > that Connect doesn't do anything with it except pass it through to the > > API. > > > It also makes clear that using "undefined" is a perfectly fine > > > implementation for PossiblyVersioned to have. > > > > > > Kind regards, > > > > > > Tom > > > > > > > > > > > > > > > > > > > > > On Thu, 27 Jan 2022 at 18:37, Chris Egerton > > > > > wrote: > > > > > > > Hi Mickael, > > > > > > > > +1 for the warning at startup, it'd be great if we could get everyone > > to > > > > start versioning all of their components. Although given the current > > state > > > > of WARN-level logging in Connect (see > > > > https://issues.apache.org/jira/browse/KAFKA-7509) there might be too > > much > > > > noise for most people to notice, unfortunately. Still, better than > > nothing! > > > > > > > > I'm not sure it'd be wise to move quite so quickly with deprecating and > > > > then removing the PossiblyVersioned interface. In an ideal world > > everyone > > > > would update their converters, SMTs, and predicates as soon as they > > saw the > > > > warning, but there could be some plugins out there that aren't > > maintained > > > > anymore and would become incompatible with a Connect runtime that > > requires > > > > them to provide versions. I can imagine there might be some fairly > > trivial > > > > but valuable SMTs or converters that were written years ago and haven't > > > > been touched in some time but are still useful to Connect users today. > > > > > > > > I think it'd be fine to log a scary-looking warning message stating > > that > > > > the plugin may be incompatible with future versions of Connect, but > > perhaps > > > > we can leave the decision about how and when we actually create that > > > > incompatibility to a future KIP. If we take the example use case of > > loading > > > > multiple versions of the same plugin on the same worker, for example, > > that > > > > feature would obviously require plugins to provide versions, but we > > could > > > > potentially add some failsafe logic in order to still support > > versionless > > > > plugins with the understanding that it would not be possible to > > colocate > > > > multiple versions of them on the same worker. > > > > > > > > Cheers, > > > > > > > > Chris > > > > > > > > On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison < > > mickael.mai...@gmail.com> > > > > wrote: > > > > > > > > > Hi Chris, > > > > > > > > > > Thanks for taking another look! > > > > > > > > > > That's a good point which I should have mentioned in the KIP. I think > > > > > we can have a PossiblyVersioned interface that extends Versioned and > > > > > has the default implementation. I'd like to also print a warning at > > > > > startup if a plugin does not override version(). Thanks for the > > > > > suggestion! > > > > > > > > > > I'm even considering whether PossiblyVersioned could be deprecated > > > > > immediately in order to be removed in the
Re: KIP-769: Connect API to retrieve connector configuration definitions
I think a simple approach here is fine. +1 for removing versioning for other plugins. On Fri, Jan 28, 2022 at 11:17 AM Mickael Maison wrote: > Thanks Chris and Tom for the feedback. > > I'm having a hard time making a decision on this point! > > Re-reading your previous comments, I agree that we're lacking a > contract on the version field, and we can see value (supporting > multiple versions for example) if this was improved. > But I'm starting to wonder if this should be handled in a different KIP. > > In this KIP, we can keep the existing Versioned interface for > Connectors and either not display a version for other plugins or > hard-code it to something like "N/A". Later if we think we're missing > the version for other plugins we can decide how to handle it. While it > would have been nice to harmonize all plugins, I don't think it's a > requirement or blocker for the rest of the value of this KIP. > > Unless you disagree and feel like we need versions for all plugins, > I'll update the KIP accordingly. > > Thanks, > Mickael > > > > > > On Thu, Jan 27, 2022 at 9:00 PM Tom Bentley wrote: > > > > To be honest I'm very sceptical that you can evolve (without breaking > > compatibility) from a situation of having a large number of plugins which > > have already implemented version() to return whatever they please, to a > > situation where you can rely on it as the basis for some feature to > support > > multiple versions of plugins. It would be much safer to introduce a new > > versioning concept, perhaps using Jar manifests and > > "Specification-Version", than to try to build on Versioned. > > > > So I would prefer Javadoc that clearly said that this version number was > a > > means for the plugin author to identify the version of their plugin in > use > > by users, for the purpose of bug reporting. That at least makes is clear > > that Connect doesn't do anything with it except pass it through to the > API. > > It also makes clear that using "undefined" is a perfectly fine > > implementation for PossiblyVersioned to have. > > > > Kind regards, > > > > Tom > > > > > > > > > > > > > > On Thu, 27 Jan 2022 at 18:37, Chris Egerton > > > wrote: > > > > > Hi Mickael, > > > > > > +1 for the warning at startup, it'd be great if we could get everyone > to > > > start versioning all of their components. Although given the current > state > > > of WARN-level logging in Connect (see > > > https://issues.apache.org/jira/browse/KAFKA-7509) there might be too > much > > > noise for most people to notice, unfortunately. Still, better than > nothing! > > > > > > I'm not sure it'd be wise to move quite so quickly with deprecating and > > > then removing the PossiblyVersioned interface. In an ideal world > everyone > > > would update their converters, SMTs, and predicates as soon as they > saw the > > > warning, but there could be some plugins out there that aren't > maintained > > > anymore and would become incompatible with a Connect runtime that > requires > > > them to provide versions. I can imagine there might be some fairly > trivial > > > but valuable SMTs or converters that were written years ago and haven't > > > been touched in some time but are still useful to Connect users today. > > > > > > I think it'd be fine to log a scary-looking warning message stating > that > > > the plugin may be incompatible with future versions of Connect, but > perhaps > > > we can leave the decision about how and when we actually create that > > > incompatibility to a future KIP. If we take the example use case of > loading > > > multiple versions of the same plugin on the same worker, for example, > that > > > feature would obviously require plugins to provide versions, but we > could > > > potentially add some failsafe logic in order to still support > versionless > > > plugins with the understanding that it would not be possible to > colocate > > > multiple versions of them on the same worker. > > > > > > Cheers, > > > > > > Chris > > > > > > On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison < > mickael.mai...@gmail.com> > > > wrote: > > > > > > > Hi Chris, > > > > > > > > Thanks for taking another look! > > > > > > > > That's a good point which I should have mentioned in the KIP. I think > > > > we can have a PossiblyVersioned interface that extends Versioned and > > > > has the default implementation. I'd like to also print a warning at > > > > startup if a plugin does not override version(). Thanks for the > > > > suggestion! > > > > > > > > I'm even considering whether PossiblyVersioned could be deprecated > > > > immediately in order to be removed in the next major release. If so, > > > > the warning message I mentioned would state that plugins not > > > > overriding version() would stop working in the next major release. > > > > > > > > Thanks, > > > > Mickael > > > > > > > > > > >
Re: KIP-769: Connect API to retrieve connector configuration definitions
Thanks Chris and Tom for the feedback. I'm having a hard time making a decision on this point! Re-reading your previous comments, I agree that we're lacking a contract on the version field, and we can see value (supporting multiple versions for example) if this was improved. But I'm starting to wonder if this should be handled in a different KIP. In this KIP, we can keep the existing Versioned interface for Connectors and either not display a version for other plugins or hard-code it to something like "N/A". Later if we think we're missing the version for other plugins we can decide how to handle it. While it would have been nice to harmonize all plugins, I don't think it's a requirement or blocker for the rest of the value of this KIP. Unless you disagree and feel like we need versions for all plugins, I'll update the KIP accordingly. Thanks, Mickael On Thu, Jan 27, 2022 at 9:00 PM Tom Bentley wrote: > > To be honest I'm very sceptical that you can evolve (without breaking > compatibility) from a situation of having a large number of plugins which > have already implemented version() to return whatever they please, to a > situation where you can rely on it as the basis for some feature to support > multiple versions of plugins. It would be much safer to introduce a new > versioning concept, perhaps using Jar manifests and > "Specification-Version", than to try to build on Versioned. > > So I would prefer Javadoc that clearly said that this version number was a > means for the plugin author to identify the version of their plugin in use > by users, for the purpose of bug reporting. That at least makes is clear > that Connect doesn't do anything with it except pass it through to the API. > It also makes clear that using "undefined" is a perfectly fine > implementation for PossiblyVersioned to have. > > Kind regards, > > Tom > > > > > > > On Thu, 27 Jan 2022 at 18:37, Chris Egerton > wrote: > > > Hi Mickael, > > > > +1 for the warning at startup, it'd be great if we could get everyone to > > start versioning all of their components. Although given the current state > > of WARN-level logging in Connect (see > > https://issues.apache.org/jira/browse/KAFKA-7509) there might be too much > > noise for most people to notice, unfortunately. Still, better than nothing! > > > > I'm not sure it'd be wise to move quite so quickly with deprecating and > > then removing the PossiblyVersioned interface. In an ideal world everyone > > would update their converters, SMTs, and predicates as soon as they saw the > > warning, but there could be some plugins out there that aren't maintained > > anymore and would become incompatible with a Connect runtime that requires > > them to provide versions. I can imagine there might be some fairly trivial > > but valuable SMTs or converters that were written years ago and haven't > > been touched in some time but are still useful to Connect users today. > > > > I think it'd be fine to log a scary-looking warning message stating that > > the plugin may be incompatible with future versions of Connect, but perhaps > > we can leave the decision about how and when we actually create that > > incompatibility to a future KIP. If we take the example use case of loading > > multiple versions of the same plugin on the same worker, for example, that > > feature would obviously require plugins to provide versions, but we could > > potentially add some failsafe logic in order to still support versionless > > plugins with the understanding that it would not be possible to colocate > > multiple versions of them on the same worker. > > > > Cheers, > > > > Chris > > > > On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison > > wrote: > > > > > Hi Chris, > > > > > > Thanks for taking another look! > > > > > > That's a good point which I should have mentioned in the KIP. I think > > > we can have a PossiblyVersioned interface that extends Versioned and > > > has the default implementation. I'd like to also print a warning at > > > startup if a plugin does not override version(). Thanks for the > > > suggestion! > > > > > > I'm even considering whether PossiblyVersioned could be deprecated > > > immediately in order to be removed in the next major release. If so, > > > the warning message I mentioned would state that plugins not > > > overriding version() would stop working in the next major release. > > > > > > Thanks, > > > Mickael > > > > > > >
Re: KIP-769: Connect API to retrieve connector configuration definitions
To be honest I'm very sceptical that you can evolve (without breaking compatibility) from a situation of having a large number of plugins which have already implemented version() to return whatever they please, to a situation where you can rely on it as the basis for some feature to support multiple versions of plugins. It would be much safer to introduce a new versioning concept, perhaps using Jar manifests and "Specification-Version", than to try to build on Versioned. So I would prefer Javadoc that clearly said that this version number was a means for the plugin author to identify the version of their plugin in use by users, for the purpose of bug reporting. That at least makes is clear that Connect doesn't do anything with it except pass it through to the API. It also makes clear that using "undefined" is a perfectly fine implementation for PossiblyVersioned to have. Kind regards, Tom On Thu, 27 Jan 2022 at 18:37, Chris Egerton wrote: > Hi Mickael, > > +1 for the warning at startup, it'd be great if we could get everyone to > start versioning all of their components. Although given the current state > of WARN-level logging in Connect (see > https://issues.apache.org/jira/browse/KAFKA-7509) there might be too much > noise for most people to notice, unfortunately. Still, better than nothing! > > I'm not sure it'd be wise to move quite so quickly with deprecating and > then removing the PossiblyVersioned interface. In an ideal world everyone > would update their converters, SMTs, and predicates as soon as they saw the > warning, but there could be some plugins out there that aren't maintained > anymore and would become incompatible with a Connect runtime that requires > them to provide versions. I can imagine there might be some fairly trivial > but valuable SMTs or converters that were written years ago and haven't > been touched in some time but are still useful to Connect users today. > > I think it'd be fine to log a scary-looking warning message stating that > the plugin may be incompatible with future versions of Connect, but perhaps > we can leave the decision about how and when we actually create that > incompatibility to a future KIP. If we take the example use case of loading > multiple versions of the same plugin on the same worker, for example, that > feature would obviously require plugins to provide versions, but we could > potentially add some failsafe logic in order to still support versionless > plugins with the understanding that it would not be possible to colocate > multiple versions of them on the same worker. > > Cheers, > > Chris > > On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison > wrote: > > > Hi Chris, > > > > Thanks for taking another look! > > > > That's a good point which I should have mentioned in the KIP. I think > > we can have a PossiblyVersioned interface that extends Versioned and > > has the default implementation. I'd like to also print a warning at > > startup if a plugin does not override version(). Thanks for the > > suggestion! > > > > I'm even considering whether PossiblyVersioned could be deprecated > > immediately in order to be removed in the next major release. If so, > > the warning message I mentioned would state that plugins not > > overriding version() would stop working in the next major release. > > > > Thanks, > > Mickael > > > >
Re: KIP-769: Connect API to retrieve connector configuration definitions
Hi Mickael, +1 for the warning at startup, it'd be great if we could get everyone to start versioning all of their components. Although given the current state of WARN-level logging in Connect (see https://issues.apache.org/jira/browse/KAFKA-7509) there might be too much noise for most people to notice, unfortunately. Still, better than nothing! I'm not sure it'd be wise to move quite so quickly with deprecating and then removing the PossiblyVersioned interface. In an ideal world everyone would update their converters, SMTs, and predicates as soon as they saw the warning, but there could be some plugins out there that aren't maintained anymore and would become incompatible with a Connect runtime that requires them to provide versions. I can imagine there might be some fairly trivial but valuable SMTs or converters that were written years ago and haven't been touched in some time but are still useful to Connect users today. I think it'd be fine to log a scary-looking warning message stating that the plugin may be incompatible with future versions of Connect, but perhaps we can leave the decision about how and when we actually create that incompatibility to a future KIP. If we take the example use case of loading multiple versions of the same plugin on the same worker, for example, that feature would obviously require plugins to provide versions, but we could potentially add some failsafe logic in order to still support versionless plugins with the understanding that it would not be possible to colocate multiple versions of them on the same worker. Cheers, Chris On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison wrote: > Hi Chris, > > Thanks for taking another look! > > That's a good point which I should have mentioned in the KIP. I think > we can have a PossiblyVersioned interface that extends Versioned and > has the default implementation. I'd like to also print a warning at > startup if a plugin does not override version(). Thanks for the > suggestion! > > I'm even considering whether PossiblyVersioned could be deprecated > immediately in order to be removed in the next major release. If so, > the warning message I mentioned would state that plugins not > overriding version() would stop working in the next major release. > > Thanks, > Mickael > > On Mon, Jan 24, 2022 at 5:41 PM Mickael Maison > wrote: > > > > Hi Tom, > > > > Thanks for the feedback. > > > > You're right we don't have any contracts around the version field. > > Like Chris I've only used the current field to find the exact source > > code of connectors. Even if it's a limited use, I think that's enough > > to justify adding it to all plugins and making them consistent. Work > > for improving the contract can be done in a separate KIP. > > > > Thanks, > > Mickael > > > > On Wed, Jan 19, 2022 at 7:55 PM Chris Egerton > > wrote: > > > > > > Hi all, > > > > > > With regards to Tom's question about possible uses of the version > string, > > > one of the most useful applications I've found for it so far has been > > > debugging connector issues, especially based on captured logs. Using > the > > > version string to identify a git tag containing the exact source code > for a > > > connector, and then using that as a reference when poring over log > files > > > that usually contain class names, method names, and/or line numbers, > is a > > > great way to construct a timeline of events and understand how things > may > > > have gone wrong. I don't think we can require this as part of the > contract > > > for the version method, but at least identifying this use case in the > docs > > > for it could be useful to help connector/SMT/converter/etc. developers > > > understand why they should care about what this method returns and how > it > > > could be beneficial to themselves and their users down the road. > > > > > > One other potential use case that comes up is that, if/when Connect > > > supports loading multiple versions of the same plugin type on a single > > > worker, the version string can be used to identify plugins that share > the > > > same name and type but which should be exposed individually to users. > If we > > > want to leave room for this possibility, perhaps part of the contract > for > > > the method can be that "any two releases of the same plugin but with > > > different behavior should return different values from their version > > > method". > > > > > > One final note--I've just realized that, with the introduction of a > default > > > implementation for the Versioned interface, we've made > Connector::version > > > and ConnectRestExtension::version optional to implement, whereas > before it > > > was required that instances of these plugins explicitly implement that > > > method. Should we allow this? An alternative could be to add a new > > > "PossiblyVersioned" interface (name obviously subject to change) that > > > extends the current Versioned interface and contains a default > > > implementation that returns the
Re: KIP-769: Connect API to retrieve connector configuration definitions
Hi Chris, Thanks for taking another look! That's a good point which I should have mentioned in the KIP. I think we can have a PossiblyVersioned interface that extends Versioned and has the default implementation. I'd like to also print a warning at startup if a plugin does not override version(). Thanks for the suggestion! I'm even considering whether PossiblyVersioned could be deprecated immediately in order to be removed in the next major release. If so, the warning message I mentioned would state that plugins not overriding version() would stop working in the next major release. Thanks, Mickael On Mon, Jan 24, 2022 at 5:41 PM Mickael Maison wrote: > > Hi Tom, > > Thanks for the feedback. > > You're right we don't have any contracts around the version field. > Like Chris I've only used the current field to find the exact source > code of connectors. Even if it's a limited use, I think that's enough > to justify adding it to all plugins and making them consistent. Work > for improving the contract can be done in a separate KIP. > > Thanks, > Mickael > > On Wed, Jan 19, 2022 at 7:55 PM Chris Egerton > wrote: > > > > Hi all, > > > > With regards to Tom's question about possible uses of the version string, > > one of the most useful applications I've found for it so far has been > > debugging connector issues, especially based on captured logs. Using the > > version string to identify a git tag containing the exact source code for a > > connector, and then using that as a reference when poring over log files > > that usually contain class names, method names, and/or line numbers, is a > > great way to construct a timeline of events and understand how things may > > have gone wrong. I don't think we can require this as part of the contract > > for the version method, but at least identifying this use case in the docs > > for it could be useful to help connector/SMT/converter/etc. developers > > understand why they should care about what this method returns and how it > > could be beneficial to themselves and their users down the road. > > > > One other potential use case that comes up is that, if/when Connect > > supports loading multiple versions of the same plugin type on a single > > worker, the version string can be used to identify plugins that share the > > same name and type but which should be exposed individually to users. If we > > want to leave room for this possibility, perhaps part of the contract for > > the method can be that "any two releases of the same plugin but with > > different behavior should return different values from their version > > method". > > > > One final note--I've just realized that, with the introduction of a default > > implementation for the Versioned interface, we've made Connector::version > > and ConnectRestExtension::version optional to implement, whereas before it > > was required that instances of these plugins explicitly implement that > > method. Should we allow this? An alternative could be to add a new > > "PossiblyVersioned" interface (name obviously subject to change) that > > extends the current Versioned interface and contains a default > > implementation that returns the "undefined" string. The to-be-updated > > plugin interfaces could extend from this new "PossiblyVersioned" interface, > > and the plugin interfaces that extend from the current "Versioned" > > interface could be left unchanged. > > > > Cheers, > > > > Chris > > > > On Wed, Jan 19, 2022 at 10:04 AM Tom Bentley wrote: > > > > > Hi Mickael, > > > > > > Thanks for the KIP. Overall I think this looks good. > > > > > > I have a small concern about the proliferation of Versioned. Currently > > > this > > > interface has a really weak contract. There's no requirement for versions > > > to be comparable, or documentation about when versions should be changed > > > by > > > authors of plugins which extend this interface. For example, if I release > > > version 1.2.3 of my SMT should the version be 1.2.3 or something else? Is > > > compatibility implied for different versions of my jar file which return > > > the same result from version()? Because of this it's not really clear what > > > the consumer of the version string is supposed to do with it. So I think > > > if > > > we're going to be using Versioned more extensively it would be good to > > > make > > > the expectations clearer, for both authors and people wanting to reason > > > about versions. > > > > > > Many thanks, > > > > > > Tom > > > > > > On Mon, 6 Dec 2021 at 09:56, Mickael Maison > > > wrote: > > > > > > > Hi Arjun, > > > > > > > > Thank you for taking a look at this KIP. > > > > > > > > 1.a Yes this is exactly what this KIP is proposing. > > > > > > > > 1.b I prefer returning all plugins together instead of having different > > > > paths. > > > > > > > > 2. While we seem to agree it would be nice to expose worker plugins, I > > > > don't think we've reached a consensus on the right way to do it. For > > > > that
Re: KIP-769: Connect API to retrieve connector configuration definitions
Hi Tom, Thanks for the feedback. You're right we don't have any contracts around the version field. Like Chris I've only used the current field to find the exact source code of connectors. Even if it's a limited use, I think that's enough to justify adding it to all plugins and making them consistent. Work for improving the contract can be done in a separate KIP. Thanks, Mickael On Wed, Jan 19, 2022 at 7:55 PM Chris Egerton wrote: > > Hi all, > > With regards to Tom's question about possible uses of the version string, > one of the most useful applications I've found for it so far has been > debugging connector issues, especially based on captured logs. Using the > version string to identify a git tag containing the exact source code for a > connector, and then using that as a reference when poring over log files > that usually contain class names, method names, and/or line numbers, is a > great way to construct a timeline of events and understand how things may > have gone wrong. I don't think we can require this as part of the contract > for the version method, but at least identifying this use case in the docs > for it could be useful to help connector/SMT/converter/etc. developers > understand why they should care about what this method returns and how it > could be beneficial to themselves and their users down the road. > > One other potential use case that comes up is that, if/when Connect > supports loading multiple versions of the same plugin type on a single > worker, the version string can be used to identify plugins that share the > same name and type but which should be exposed individually to users. If we > want to leave room for this possibility, perhaps part of the contract for > the method can be that "any two releases of the same plugin but with > different behavior should return different values from their version > method". > > One final note--I've just realized that, with the introduction of a default > implementation for the Versioned interface, we've made Connector::version > and ConnectRestExtension::version optional to implement, whereas before it > was required that instances of these plugins explicitly implement that > method. Should we allow this? An alternative could be to add a new > "PossiblyVersioned" interface (name obviously subject to change) that > extends the current Versioned interface and contains a default > implementation that returns the "undefined" string. The to-be-updated > plugin interfaces could extend from this new "PossiblyVersioned" interface, > and the plugin interfaces that extend from the current "Versioned" > interface could be left unchanged. > > Cheers, > > Chris > > On Wed, Jan 19, 2022 at 10:04 AM Tom Bentley wrote: > > > Hi Mickael, > > > > Thanks for the KIP. Overall I think this looks good. > > > > I have a small concern about the proliferation of Versioned. Currently this > > interface has a really weak contract. There's no requirement for versions > > to be comparable, or documentation about when versions should be changed by > > authors of plugins which extend this interface. For example, if I release > > version 1.2.3 of my SMT should the version be 1.2.3 or something else? Is > > compatibility implied for different versions of my jar file which return > > the same result from version()? Because of this it's not really clear what > > the consumer of the version string is supposed to do with it. So I think if > > we're going to be using Versioned more extensively it would be good to make > > the expectations clearer, for both authors and people wanting to reason > > about versions. > > > > Many thanks, > > > > Tom > > > > On Mon, 6 Dec 2021 at 09:56, Mickael Maison > > wrote: > > > > > Hi Arjun, > > > > > > Thank you for taking a look at this KIP. > > > > > > 1.a Yes this is exactly what this KIP is proposing. > > > > > > 1.b I prefer returning all plugins together instead of having different > > > paths. > > > > > > 2. While we seem to agree it would be nice to expose worker plugins, I > > > don't think we've reached a consensus on the right way to do it. For > > > that reason, I'm dropping this feature from this KIP. We can address > > > this issue in another KIP. I have updated the KIP. > > > > > > Thanks, > > > Mickael > > > > > > On Mon, Dec 6, 2021 at 10:15 AM Mickael Maison > > > > > wrote: > > > > > > > > Thanks Randall, updated. > > > > > > > > On Sun, Dec 5, 2021 at 6:05 AM Arjun Satish > > > wrote: > > > > > > > > > > hey folks, > > > > > > > > > > Thanks a lot for the KIP and the discussions. Here are a couple of > > > thoughts > > > > > (apologies if I missed some point in this thread). > > > > > > > > > > 1. it seems like we are changing the behavior of /connector-plugins > > by > > > now > > > > > returning the list of transforms, converters as well? This would > > break > > > > > backward compatibility, and clients would need to track what version > > of > > > > > Connect workers they are querying to keep their
Re: KIP-769: Connect API to retrieve connector configuration definitions
> and > > > > > JSON schema > > > > > > > > > > > > > > converters > > > > > > > > > > > > > > > would benefit from this kind of feature. It's a > > little > > > > > tangential to > > > > > > > > > > > > this > > > > > > > > > > > > > > > KIP (which at the moment is about discovering > > plugins > > > > > and their > > > > > > > > > > > > > > > configuration surfaces, as opposed to > validating > > > > > them), but I > > > > > > > > > > > > figured I'd > > > > > > > > > > > > > > > ask since we're going to be expanding the > > Converter > > > > > interface and it > > > > > > > > > > > > may > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > useful to tackle this while we're in the > > neighborhood. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4. The description for the new > > > > > /plugins///configdef > > > > > > > > > > > > endpoint > > > > > > > > > > > > > > > states that "Name must be the fully qualified > > class > > > > > name of the > > > > > > > > > > > > plugin". > > > > > > > > > > > > > > > Any reason not to also support aliases (e.g., > > > > > > > > > > > > "FileStreamSinkConnector" > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > "FileStreamSink" instead of > > > > > > > > > > > > > > > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Chris > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael > Maison < > > > > > > > > > > > > > > mickael.mai...@gmail.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks all for the feedback! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Chris, > > > > > > > > > > > > > > > > I agree that fixing the current endpoint > helps > > a > > > > > lot. Thanks for > > > > > > > > > > > > > > > > raising these JIRAs and submitting a PR! > > > > > > > > > > > > > > > > However thinking about the issue further, I > > decided > > > > > to expand the > > > > > > > > > > > > > > > > scope of the KIP to cover all user-visible > > plugins. > > > > > > > > > > > > > > > > In practice, users want to know about all > > available > > > > > plugins not > > > > > > > > > > > > only > > > > > > > > > > > > > > > > connectors. This includes transformations, > > > > > converters, > > > > > > > > > > > > > > > > header_converters and predicates. As we also > > want to > > > > > retrieve > > > > > > > > > > > > > > > > configdef for these too, I think it makes > > sense to > > > > > introduce a new > > > > > > > > > > > > > > > > endpoint to do
Re: KIP-769: Connect API to retrieve connector configuration definitions
t; > > useful to tackle this while we're in the > neighborhood. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4. The description for the new > > > > /plugins///configdef > > > > > > > > > > > endpoint > > > > > > > > > > > > > > states that "Name must be the fully qualified > class > > > > name of the > > > > > > > > > > > plugin". > > > > > > > > > > > > > > Any reason not to also support aliases (e.g., > > > > > > > > > > > "FileStreamSinkConnector" > > > > > > > > > > > > > or > > > > > > > > > > > > > > "FileStreamSink" instead of > > > > > > > > > > > > > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Chris > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison < > > > > > > > > > > > > > mickael.mai...@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks all for the feedback! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Chris, > > > > > > > > > > > > > > > I agree that fixing the current endpoint helps > a > > > > lot. Thanks for > > > > > > > > > > > > > > > raising these JIRAs and submitting a PR! > > > > > > > > > > > > > > > However thinking about the issue further, I > decided > > > > to expand the > > > > > > > > > > > > > > > scope of the KIP to cover all user-visible > plugins. > > > > > > > > > > > > > > > In practice, users want to know about all > available > > > > plugins not > > > > > > > > > > > only > > > > > > > > > > > > > > > connectors. This includes transformations, > > > > converters, > > > > > > > > > > > > > > > header_converters and predicates. As we also > want to > > > > retrieve > > > > > > > > > > > > > > > configdef for these too, I think it makes > sense to > > > > introduce a new > > > > > > > > > > > > > > > endpoint to do so. Alongside we obviously need > a new > > > > endpoint for > > > > > > > > > > > > > > > listing all plugins. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Gunnar, > > > > > > > > > > > > > > > I took a look at exposing valid values via the > API. > > > > I think the > > > > > > > > > > > issue > > > > > > > > > > > > > > > is that Validators don't expose a way to > retrieve > > > > valid values. > > > > > > > > > > > > > > > Changing validators will have an impact on all > > > > components so I'd > > > > > > > > > > > > > > > prefer to address this requirement in a > separate > > > > KIP. I agree this > > > > > > > > > > > > > > > would be an interesting improvement and I'd > happy to > > > > write a KIP > > > > > > > > > > > for > > > > > > > > > > > > > > > it too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > &
Re: KIP-769: Connect API to retrieve connector configuration definitions
ess this requirement in a separate > > > KIP. I agree this > > > > > > > > > > > > > > would be an interesting improvement and I'd happy to > > > write a KIP > > > > > > > > > > for > > > > > > > > > > > > > > it too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have updated the KIP accordingly. Let me know if > > > you have further > > > > > > > > > > > > > > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Mickael > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining > > > config > > > > > > > > > > definitions. It > > > > > > > > > > > > > > > always felt odd to me that one has to issue a PUT > > > for that > > > > > > > > > > purpose. > > > > > > > > > > > > If > > >
Re: KIP-769: Connect API to retrieve connector configuration definitions
gt; > > > > > > > > > > > header_converters and predicates. As we also want to > > retrieve > > > > > > > > > > > > > configdef for these too, I think it makes sense to > > introduce a new > > > > > > > > > > > > > endpoint to do so. Alongside we obviously need a new > > endpoint for > > > > > > > > > > > > > listing all plugins. > > > > > > > > > > > > > > > > > > > > > > > > > > Gunnar, > > > > > > > > > > > > > I took a look at exposing valid values via the API. > > I think the > > > > > > > > > issue > > > > > > > > > > > > > is that Validators don't expose a way to retrieve > > valid values. > > > > > > > > > > > > > Changing validators will have an impact on all > > components so I'd > > > > > > > > > > > > > prefer to address this requirement in a separate > > KIP. I agree this > > > > > > > > > > > > > would be an interesting improvement and I'd happy to > > write a KIP > > > > > > > > > for > > > > > > > > > > > > > it too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have updated the KIP accordingly. Let me know if > > you have further > > > > > > > > > > > > > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Mickael > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining > > config > > > > > > > > > definitions. It > > > > > > > > > > > > > > always felt odd to me that one has to issue a PUT > > for that > > > > > > > > > purpose. > > > > > > > > > > > If > > > > > > > > > > > > > > nothing else, it'd be better in terms of > > discoverability of the > > > > > > > > > KC > > > > > > > > > > > REST > > > > > > > > > > > > > API. > > > > > > > > > > > > > > > > > > > > > > > > > > > > One additional feature request I'd have is to > > expose the valid > > > > > > > > > enum > > > > > > > > > > > > > > constants for enum-typed options. That'll help to > > display the > > > > > > > > > values > > > > > > > > > > > in a > > > > > > > > > > > > > > drop-down or via radio buttons in a UI, give us > > tab completion in > > > > > > > > > > > kcctl, > > > > > > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > > > --
Re: KIP-769: Connect API to retrieve connector configuration definitions
t; > > > > > > > > > > > API.
> > > > > > > > > > > > >
> > > > > > > > > > > > > One additional feature request I'd have is to
> expose the valid
> > > > > > > > enum
> > > > > > > > > > > > > constants for enum-typed options. That'll help to
> display the
> > > > > > > > values
> > > > > > > > > > in a
> > > > > > > > > > > > > drop-down or via radio buttons in a UI, give us
> tab completion in
> > > > > > > > > > kcctl,
> > > > > > > > > > > > > etc.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > >
> > > > > > > > > > > > > --Gunnar
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris
> Egerton
> > > > > > > > > > > > > :
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It sounds like there are three major points here
> in favor of a
> > > > > > > > new
> > > > > > > > > > GET
> > > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request
> for sink
> > > > > > > > connectors
> > > > > > > > > > > > because a
> > > > > > > > > > > > > > topic li
Re: KIP-769: Connect API to retrieve connector configuration definitions
t; > > > > > > > > > >
> > > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > > > connectors
> > > > > > > > > > > because a
> > > > > > > > > > > > > topic list/topic regex has to be supplied (otherwise
> > > > > > > > > > > > > the PUT
> > > > > > > > > endpoint
> > > > > > > > > > > > > returns a 500 response)
&
Re: KIP-769: Connect API to retrieve connector configuration definitions
t; > etc.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > --Gunnar
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > > > > :
> > > > > > > > > > >
> > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > >
> > > > > > > > > > > > It sounds like there are three major points here in
> > favor of a
> > > > > > new
> > > > > > > > GET
> > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > > connectors
> > > > > > > > > > because a
> > > > > > > > > > > > topic list/topic regex has to be supplied (otherwise
> > the PUT
> > > > > > > > endpoint
> > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > 2. A dummy request still triggers custom validations
> > by the
> > > > > > > > connector,
> > > > > > > > > > > > which may be best to avoid if we know for sure that
> > the config
> > > > > > > > isn't
> > > > > > > > > > worth
> > > > > > > > > > > > validating yet
> > > > > > > > > > > > 3. It's more ergonomic and intuitive to be able to
> > issue a GET
> > > > > > > > request
> > > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > > >
> > > > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327)
> > with a fix
> > > > > > > > already
> > > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think
> > it'd be
> > > > > > > > better to
> > > > > > > > > > > > focus on fixing this bug in general instead of
> > implementing a
> > > > > > new
> > > > > > > > REST
> > > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > > >
> > > > > > > > > > > > With regards to 2, this is technically possible but
> > I'm unsure
> > > > > > > > it'd be
> > > > > > > > > > too
> > > > > > > > > > > > common out in the wild given that most validations
> > that could
> > > > > > be
> > > > > > > > > > expensive
> > > > > > > > > > > > would involve things like connecting to a database,
> > checking
> > > > > > if a
> > > > > > > > cloud
> > > > > > > > > > > > storage bucket exists, etc., none of which are
> > possible without
> > > > > > > > some
> > > > > > > > > > > > configuration properties from the user (db hostname,
> > bucket
> > > > > > name,
> > > > > > > > > > etc.).
> > > > > > > > > > > >
> > > > > > > > > > > > With regards to 3, I do agree that it'd be easier for
> > people
> > > > > > > > designing
> > > > > > > > > > UIs
> > > > > > > > > > > > to have a GET API to work against. I'm just not sure
> > it's
> > > > > > worth the
> > > > > > > > > > > > additional implementation, testing, and maintenance
> > burden. If
> > > > > > it
> > > > > > > > were
> > > > > > > > > > > > possible to issue a PUT request without unexpected
> > 500s for
> > > > > > invalid
> > > > > > > > > > > > configs, would that suffice? AFAICT it'd basically be
> > as
> > > > > > simple as
> > > > > > > > > > issuing
> > > > > > > > > > > > a PUT request with a dummy body consisting of nothing
> > except
> > > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > > class (which at this point we might even make
> > unnecessary and
> > > > > > just
> > > > > > > > > > > > automatically replace with the connector class from
> > the URL)
> > > > > > and
> > > > > > > > then
> > > > > > > > > > > > filtering the response to just grab the "definition"
> > field of
> > > > > > each
> > > > > > > > > > element
> > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > > > viktorsomo...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I too think this would be a very useful feature.
> > Some of our
> > > > > > > > > > management
> > > > > > > > > > > > > applications would provide a wizard for creating
> > connectors.
> > > > > > In
> > > > > > > > this
> > > > > > > > > > > > > scenario the user basically would fill out a sample
> > > > > > configuration
> > > > > > > > > > > > generated
> > > > > > > > > > > > > by the UI which would send it back to Connect for
> > validation
> > > > > > and
> > > > > > > > > > > > eventually
> > > > > > > > > > > > > create a new connector. The first part of this
> > workflow can
> > > > > > be
> > > > > > > > > > enhanced
> > > > > > > > > > > > if
> > > > > > > > > > > > > we had an API that can return the configuration
> > definition
> > > > > > of the
> > > > > > > > > > given
> > > > > > > > > > > > > type of connector as the UI application would be
> > able to
> > > > > > > > generate a
> > > > > > > > > > > > sample
> > > > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > The
> > connector-plugins/{connectorType}/config/validate API
> > > > > > > > essentially
> > > > > > > > > > > > works
> > > > > > > > > > > > > and returns the data that we need, however it is a
> > HTTP PUT
> > > > > > API
> > > > > > > > that
> > > > > > > > > > is a
> > > > > > > > > > > > > bit unintuitive for a fetch-like functionality and
> > also
> > > > > > > > functionally
> > > > > > > > > > > > > different as it validates the given (dummy) request.
> > In case
> > > > > > of
> > > > > > > > sink
> > > > > > > > > > > > > connectors one would need to also provide a topic
> > name.
> > > > > > > > > > > > >
> > > > > > > > > > > > > A suggestion for the KIP: I think it can be useful
> > to return
> > > > > > the
> > > > > > > > > > config
> > > > > > > > > > > > > groups and the connector class' name similarly to the
> > > > > > validate
> > > > > > > > API
> > > > > > > > > > just
> > > > > > > > > > > > in
> > > > > > > > > > > > > case any frontend needs them (and also the response
> > would be
> > > > > > more
> > > > > > > > > > like
> > > > > > > > > > > > the
> > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Viktor
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > > ryannedo...@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw.
> > Could be
> > > > > > the
> > > > > > > > same
> > > > > > > > > > > > > handler
> > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You're right, you can achieve the same
> > functionality
> > > > > > using
> > > > > > > > the
> > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > In my mind it was only for validation once you
> > have
> > > > > > build a
> > > > > > > > > > > > > > > configuration but when used with an empty
> > configuration,
> > > > > > it
> > > > > > > > > > basically
> > > > > > > > > > > > > > > serves the same purpose as the proposed new
> > endpoint.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint
> > but I
> > > > > > don't
> > > > > > > > > > think it
> > > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm wondering about the use case here. The
> > motivation
> > > > > > > > section
> > > > > > > > > > > > states
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > > > configurations
> > > > > > > > a
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > > > connector
> > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > in the worst case, look directly at the
> > connector
> > > > > > source
> > > > > > > > > > code.",
> > > > > > > > > > > > and
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > with this KIP, "users will be able to discover
> > the
> > > > > > required
> > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > for connectors installed in a Connect cluster"
> > and
> > > > > > "tools
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > able
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > > > connectors".
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > >
> > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > endpoint not address these points? What will
> > the
> > > > > > > > newly-proposed
> > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > allow users to do that they will not already
> > be able
> > > > > > to do
> > > > > > > > > > with the
> > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison
> > <
> > > > > > > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > > > configuration
> > > > > > > > > > > > definitions
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Please take a look and let me know if you
> > have any
> > > > > > > > feedback.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> >
Re: KIP-769: Connect API to retrieve connector configuration definitions
tions. That'll help to display
> > > > > > > > > > > the
> > > > > > values
> > > > > > > > in a
> > > > > > > > > > > drop-down or via radio buttons in a UI, give us tab
> > > > > > > > > > > completion in
> > > > > > > > kcctl,
> > > > > > > > > > > etc.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > --Gunnar
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > > > > :
> > > > > > > > > > >
> > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > >
> > > > > > > > > > > > It sounds like there are three major points here in
> > > > > > > > > > > > favor of a
> > > > > > new
> > > > > > > > GET
> > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > > connectors
> > > > > > > > > > because a
> > > > > > > > > > > > topic list/topic regex has to be supplied (otherwise
> > > > > > > > > > > > the PUT
> > > > > > > > endpoint
> > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > 2. A dummy request still triggers custom validations by
> > > > > > > > > > > > the
> > > > > > > > connector,
> > > > > > > > > > > > which may be best to avoid if we know for sure that the
> > > > > > > > > > > > config
> > > > > > > > isn't
> > > > > > > > > > worth
> > > > > > > > > > > > validating yet
> > > > > > > > > > > > 3. It's more ergonomic and intuitive to be able to
> > > > > > > > > > > > issue a GET
> > > > > > > > request
> > > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > > >
> > > > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > > > https://issu
Re: KIP-769: Connect API to retrieve connector configuration definitions
; > https://github.com/apache/kafka/pull/11369). I think it'd
> > > > > > > > > > > be
> > > > > > > better to
> > > > > > > > > > > focus on fixing this bug in general instead of
> > > > > > > > > > > implementing a
> > > > > new
> > > > > > > REST
> > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > >
> > > > > > > > > > > With regards to 2, this is technically possible but I'm
> > > > > > > > > > > unsure
> > > > > > > it'd be
> > > > > > > > > too
> > > > > > > > > > > common out in the wild given that most validations that
> > > > > > > > > > > could
> > > > > be
> > > > > > > > > expensive
> > > > > > > > > > > would involve things like connecting to a database,
> > > > > > > > > > > checking
> > > > > if a
> > > > > > > cloud
> > > > > > > > > > > storage bucket exists, etc., none of which are possible
> > > > > > > > > > > without
> > > > > > > some
> > > > > > > > > > > configuration properties from the user (db hostname,
> > > > > > > > > > > bucket
> > > > > name,
> > > > > > > > > etc.).
> > > > > > > > > > >
> > > > > > > > > > > With regards to 3, I do agree that it'd be easier for
> > > > > > > > > > > people
> > > > > > > designing
> > > > > > > > > UIs
> > > > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > > > worth the
> > > > > > > > > > > additional implementation, testing, and maintenance
> > > > > > > > > > > burden. If
> > > > > it
> > > > > > > were
> > > > > > > > > > > possible to issue a PUT request without unexpected 500s
> > > > > > > > > > > for
> > > > > invalid
> > > > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > > > simple as
> > > > > > > > > issuing
> > > > > > > > > > > a PUT request with a dummy body consisting of nothing
> > > > > > > > > > > except
> > > > > the
> > > > > > > > > connector
> > > > > > > > > > > class (which at this point we might even make unnecessary
> > > > > > > > > > > and
> > > > > just
> > > > > > > > > > > automatically replace with the connector class from the
> > > > > > > > > > > URL)
> > > > > and
> > > > > > > then
> > > > > > > > > > > filtering the response to just grab the "definition"
> > > > > > > > > > > field of
> > > > > each
> > > > > > > > > element
> > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > > viktorsomo...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > >
> > > > > > > > > > > > I too think this would be a very useful feature. Some
> > > > > > > > > > > > of our
> > > > > > > > > management
> > > > > > > > > > > > applications would provide a wizard for creating
> > > > > > > > > > > > connectors.
> > > > > In
> > > > > > > this
> > > > > > > > > > > > scenario the user basically would fill out a sample
> > > > > configuration
> > > > > > > > > > > generated
> > > > > > > > > > > > by the UI which would send it back to Connect for
> > > > > > > > > > > > validation
> > > > > and
> > > > > > > > > > > eventually
> > > > > > > > > > > > create a new connector. The first part of this workflow
> > > > > > > > > > > > can
> > > > > be
> > > > > > > > > enhanced
> > > > > > > > > > > if
> > > > > > > > > > > > we had an API that can return the configuration
> > > > > > > > > > > > definition
> > > > > of the
> > > > > > > > > given
> > > > > > > > > > > > type of connector as the UI application would be able to
> > > > > > > generate a
> > > > > > > > > > > sample
> > > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > The connector-plugins/{connectorType}/config/validate
> > > > > > > > > > > > API
> > > > > > > essentially
> > > > > > > > > > > works
> > > > > > > > > > > > and returns the data that we need, however it is a HTTP
> > > > > > > > > > > > PUT
> > > > > API
> > > > > > > that
> > > > > > > > > is a
> > > > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > > > functionally
> > > > > > > > > > > > different as it validates the given (dummy) request. In
> > > > > > > > > > > > case
> > > > > of
> > > > > > > sink
> > > > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > > > >
> > > > > > > > > > > > A suggestion for the KIP: I think it can be useful to
> > > > > > > > > > > > return
> > > > > the
> > > > > > > > > config
> > > > > > > > > > > > groups and the connector class' name similarly to the
> > > > > validate
> > > > > > > API
> > > > > > > > > just
> > > > > > > > > > > in
> > > > > > > > > > > > case any frontend needs them (and also the response
> > > > > > > > > > > > would be
> > > > > more
> > > > > > > > > like
> > > > > > > > > > > the
> > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > >
> > > > > > > > > > > > Viktor
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > ryannedo...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw.
> > > > > > > > > > > > > Could be
> > > > > the
> > > > > > > same
> > > > > > > > > > > > handler
> > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You're right, you can achieve the same functionality
> > > > > using
> > > > > > > the
> > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > In my mind it was only for validation once you have
> > > > > build a
> > > > > > > > > > > > > > configuration but when used with an empty
> > > > > > > > > > > > > > configuration,
> > > > > it
> > > > > > > > > basically
> > > > > > > > > > > > > > serves the same purpose as the proposed new
> > > > > > > > > > > > > > endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but
> > > > > > > > > > > > > > I
> > > > > don't
> > > > > > > > > think it
> > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm wondering about the use case here. The
> > > > > > > > > > > > > > > motivation
> > > > > > > section
> > > > > > > > > > > states
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > > configurations
> > > > > > > a
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > > connector
> > > > > > > > > > > > documentation
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > in the worst case, look directly at the connector
> > > > > source
> > > > > > > > > code.",
> > > > > > > > > > > and
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > > > required
> > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > > > "tools
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > able
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > > connectors".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > > > newly-proposed
> > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > allow users to do that they will not already be
> > > > > > > > > > > > > > > able
> > > > > to do
> > > > > > > > > with the
> > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > > configuration
> > > > > > > > > > > definitions
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Please take a look and let me know if you have
> > > > > > > > > > > > > > > > any
> > > > > > > feedback.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
Re: KIP-769: Connect API to retrieve connector configuration definitions
fix
> > > > > > > already
> > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd
> > > > > > > > > > > be
> > > > > > > better to
> > > > > > > > > > > focus on fixing this bug in general instead of
> > > > > > > > > > > implementing a
> > > > > new
> > > > > > > REST
> > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > >
> > > > > > > > > > > With regards to 2, this is technically possible but I'm
> > > > > > > > > > > unsure
> > > > > > > it'd be
> > > > > > > > > too
> > > > > > > > > > > common out in the wild given that most validations that
> > > > > > > > > > > could
> > > > > be
> > > > > > > > > expensive
> > > > > > > > > > > would involve things like connecting to a database,
> > > > > > > > > > > checking
> > > > > if a
> > > > > > > cloud
> > > > > > > > > > > storage bucket exists, etc., none of which are possible
> > > > > > > > > > > without
> > > > > > > some
> > > > > > > > > > > configuration properties from the user (db hostname,
> > > > > > > > > > > bucket
> > > > > name,
> > > > > > > > > etc.).
> > > > > > > > > > >
> > > > > > > > > > > With regards to 3, I do agree that it'd be easier for
> > > > > > > > > > > people
> > > > > > > designing
> > > > > > > > > UIs
> > > > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > > > worth the
> > > > > > > > > > > additional implementation, testing, and maintenance
> > > > > > > > > > > burden. If
> > > > > it
> > > > > > > were
> > > > > > > > > > > possible to issue a PUT request without unexpected 500s
> > > > > > > > > > > for
> > > > > invalid
> > > > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > > > simple as
> > > > > > > > > issuing
> > > > > > > > > > > a PUT request with a dummy body consisting of nothing
> > > > > > > > > > > except
> > > > > the
> > > > > > > > > connector
> > > > > > > > > > > class (which at this point we might even make unnecessary
> > > > > > > > > > > and
> > > > > just
> > > > > > > > > > > automatically replace with the connector class from the
> > > > > > > > > > > URL)
> > > > > and
> > > > > > > then
> > > > > > > > > > > filtering the response to just grab the "definition"
> > > > > > > > > > > field of
> > > > > each
> > > > > > > > > element
> > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > > viktorsomo...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > >
> > > > > > > > > > > > I too think this would be a very useful feature. Some
> > > > > > > > > > > > of our
> > > > > > > > > management
> > > > > > > > > > > > applications would provide a wizard for creating
> > > > > > > > > > > > connectors.
> > > > > In
> > > > > > > this
> > > > > > > > > > > > scenario the user basically would fill out a sample
> > > > > configuration
> > > > > > > > > > > generated
> > > > > > > > > > > > by the UI which would send it back to Connect for
> > > > > > > > > > > > validation
> > > > > and
> > > > > > > > > > > eventually
> > > > > > > > > > > > create a new connector. The first part of this workflow
> > > > > > > > > > > > can
> > > > > be
> > > > > > > > > enhanced
> > > > > > > > > > > if
> > > > > > > > > > > > we had an API that can return the configuration
> > > > > > > > > > > > definition
> > > > > of the
> > > > > > > > > given
> > > > > > > > > > > > type of connector as the UI application would be able to
> > > > > > > generate a
> > > > > > > > > > > sample
> > > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > The connector-plugins/{connectorType}/config/validate
> > > > > > > > > > > > API
> > > > > > > essentially
> > > > > > > > > > > works
> > > > > > > > > > > > and returns the data that we need, however it is a HTTP
> > > > > > > > > > > > PUT
> > > > > API
> > > > > > > that
> > > > > > > > > is a
> > > > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > > > functionally
> > > > > > > > > > > > different as it validates the given (dummy) request. In
> > > > > > > > > > > > case
> > > > > of
> > > > > > > sink
> > > > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > > > >
> > > > > > > > > > > > A suggestion for the KIP: I think it can be useful to
> > > > > > > > > > > > return
> > > > > the
> > > > > > > > > config
> > > > > > > > > > > > groups and the connector class' name similarly to the
> > > > > validate
> > > > > > > API
> > > > > > > > > just
> > > > > > > > > > > in
> > > > > > > > > > > > case any frontend needs them (and also the response
> > > > > > > > > > > > would be
> > > > > more
> > > > > > > > > like
> > > > > > > > > > > the
> > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > >
> > > > > > > > > > > > Viktor
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > ryannedo...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw.
> > > > > > > > > > > > > Could be
> > > > > the
> > > > > > > same
> > > > > > > > > > > > handler
> > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You're right, you can achieve the same functionality
> > > > > using
> > > > > > > the
> > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > In my mind it was only for validation once you have
> > > > > build a
> > > > > > > > > > > > > > configuration but when used with an empty
> > > > > > > > > > > > > > configuration,
> > > > > it
> > > > > > > > > basically
> > > > > > > > > > > > > > serves the same purpose as the proposed new
> > > > > > > > > > > > > > endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but
> > > > > > > > > > > > > > I
> > > > > don't
> > > > > > > > > think it
> > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm wondering about the use case here. The
> > > > > > > > > > > > > > > motivation
> > > > > > > section
> > > > > > > > > > > states
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > > configurations
> > > > > > > a
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > > connector
> > > > > > > > > > > > documentation
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > in the worst case, look directly at the connector
> > > > > source
> > > > > > > > > code.",
> > > > > > > > > > > and
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > > > required
> > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > > > "tools
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > able
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > > connectors".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > > > newly-proposed
> > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > allow users to do that they will not already be
> > > > > > > > > > > > > > > able
> > > > > to do
> > > > > > > > > with the
> > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > > configuration
> > > > > > > > > > > definitions
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Please take a look and let me know if you have
> > > > > > > > > > > > > > > > any
> > > > > > > feedback.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
Re: KIP-769: Connect API to retrieve connector configuration definitions
r adding a GET endpoint for obtaining config
> > > > > definitions. It
> > > > > > > > > > always felt odd to me that one has to issue a PUT for
> that
> > > > > purpose.
> > > > > > > If
> > > > > > > > > > nothing else, it'd be better in terms of discoverability
> of the
> > > > > KC
> > > > > > > REST
> > > > > > > > > API.
> > > > > > > > > >
> > > > > > > > > > One additional feature request I'd have is to expose the
> valid
> > > > > enum
> > > > > > > > > > constants for enum-typed options. That'll help to
> display the
> > > > > values
> > > > > > > in a
> > > > > > > > > > drop-down or via radio buttons in a UI, give us tab
> completion in
> > > > > > > kcctl,
> > > > > > > > > > etc.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > --Gunnar
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > > > :
> > > > > > > > > >
> > > > > > > > > > > Hi Viktor,
> > > > > > > > > > >
> > > > > > > > > > > It sounds like there are three major points here in
> favor of a
> > > > > new
> > > > > > > GET
> > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > >
> > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > connectors
> > > > > > > > > because a
> > > > > > > > > > > topic list/topic regex has to be supplied (otherwise
> the PUT
> > > > > > > endpoint
> > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > 2. A dummy request still triggers custom validations
> by the
> > > > > > > connector,
> > > > > > > > > > > which may be best to avoid if we know for sure that
> the config
> > > > > > > isn't
> > > > > > > > > worth
> > > > > > > > > > > validating yet
> > > > > > > > > > > 3. It's more ergonomic and intuitive to be able to
> issue a GET
> > > > > > > request
> > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > >
> > > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327)
> with a fix
> > > > > > > already
> > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think
> it'd be
> > > > > > > better to
> > > > > > > > > > > focus on fixing this bug in general instead of
> implementing a
> > > > > new
> > > > > > > REST
> > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > >
> > > > > > > > > > > With regards to 2, this is technically possible but
> I'm unsure
> > > > > > > it'd be
> > > > > > > > > too
> > > > > > > > > > > common out in the wild given that most validations
> that could
> > > > > be
> > > > > > > > > expensive
> > > > > > > > > > > would involve things like connecting to a database,
> checking
> > > > > if a
> > > > > > > cloud
> > > > > > > > > > > storage bucket exists, etc., none of which are
> possible without
> > > > > > > some
> > > > >
Re: KIP-769: Connect API to retrieve connector configuration definitions
> > endpoint
> > > > > > > > > > returns a 500 response)
> > > > > > > > > > 2. A dummy request still triggers custom validations by the
> > > > > > connector,
> > > > > > > > > > which may be best to avoid if we know for sure that the
> > > > > > > > > > config
> > > > > > isn't
> > > > > > > > worth
> > > > > > > > > > validating yet
> > > > > > > > > > 3. It's more ergonomic and intuitive to be able to issue a
> > > > > > > > > > GET
> > > > > > request
> > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > >
> > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a
> > > > > > > > > > fix
> > > > > > already
> > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> > > > > > better to
> > > > > > > > > > focus on fixing this bug in general instead of implementing
> > > > > > > > > > a
> > > > new
> > > > > > REST
> > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > >
> > > > > > > > > > With regards to 2, this is technically possible but I'm
> > > > > > > > > > unsure
> > > > > > it'd be
> > > > > > > > too
> > > > > > > > > > common out in the wild given that most validations that
> > > > > > > > > > could
> > > > be
> > > > > > > > expensive
> > > > > > > > > > would involve things like connecting to a database, checking
> > > > if a
> > > > > > cloud
> > > > > > > > > > storage bucket exists, etc., none of which are possible
> > > > > > > > > > without
> > > > > > some
> > > > > > > > > > configuration properties from the user (db hostname, bucket
> > > > name,
> > > > > > > > etc.).
> > > > > > > > > >
> > > > > > > > > > With regards to 3, I do agree that it'd be easier for people
> > > > > > designing
> > > > > > > > UIs
> > > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > > worth the
> > > > > > > > > > additional implementation, testing, and maintenance burden.
> > > > > > > > > > If
> > > > it
> > > > > > were
> > > > > > > > > > possible to issue a PUT request without unexpected 500s for
> > > > invalid
> > > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > > simple as
> > > > > > > > issuing
> > > > > > > > > > a PUT request with a dummy body consisting of nothing except
> > > > the
> > > > > > > > connector
> > > > > > > > > > class (which at this point we might even make unnecessary
> > > > > > > > > > and
> > > > just
> > > > > > > > > > automatically replace with the connector class from the URL)
> > > > and
> > > > > > then
> > > > > > > > > > filtering the response to just grab the "definition" field
> > > > > > > > > > of
> > > > each
> > > > > > > > element
> > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > viktorsomo...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Folks,
> > > > > > > > > > >
> > > > > > > > > > > I too think this would be a very useful feature. Some of
> > > > > > > > > > > our
> > > > > > > > management
> > > > > > > > > > > applications would provide a wizard for creating
> > > > > > > > > > > connectors.
> > > > In
> > > > > > this
> > > > > > > > > > > scenario the user basically would fill out a sample
> > > > configuration
> > > > > > > > > > generated
> > > > > > > > > > > by the UI which would send it back to Connect for
> > > > > > > > > > > validation
> > > > and
> > > > > > > > > > eventually
> > > > > > > > > > > create a new connector. The first part of this workflow
> > > > > > > > > > > can
> > > > be
> > > > > > > > enhanced
> > > > > > > > > > if
> > > > > > > > > > > we had an API that can return the configuration definition
> > > > of the
> > > > > > > > given
> > > > > > > > > > > type of connector as the UI application would be able to
> > > > > > generate a
> > > > > > > > > > sample
> > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > > essentially
> > > > > > > > > > works
> > > > > > > > > > > and returns the data that we need, however it is a HTTP
> > > > > > > > > > > PUT
> > > > API
> > > > > > that
> > > > > > > > is a
> > > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > > functionally
> > > > > > > > > > > different as it validates the given (dummy) request. In
> > > > > > > > > > > case
> > > > of
> > > > > > sink
> > > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > > >
> > > > > > > > > > > A suggestion for the KIP: I think it can be useful to
> > > > > > > > > > > return
> > > > the
> > > > > > > > config
> > > > > > > > > > > groups and the connector class' name similarly to the
> > > > validate
> > > > > > API
> > > > > > > > just
> > > > > > > > > > in
> > > > > > > > > > > case any frontend needs them (and also the response would
> > > > > > > > > > > be
> > > > more
> > > > > > > > like
> > > > > > > > > > the
> > > > > > > > > > > validate API but simpler).
> > > > > > > > > > >
> > > > > > > > > > > Viktor
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > ryannedo...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could
> > > > > > > > > > > > be
> > > > the
> > > > > > same
> > > > > > > > > > > handler
> > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > >
> > > > > > > > > > > > > You're right, you can achieve the same functionality
> > > > using
> > > > > > the
> > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > In my mind it was only for validation once you have
> > > > build a
> > > > > > > > > > > > > configuration but when used with an empty
> > > > > > > > > > > > > configuration,
> > > > it
> > > > > > > > basically
> > > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > > don't
> > > > > > > > think it
> > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm wondering about the use case here. The
> > > > > > > > > > > > > > motivation
> > > > > > section
> > > > > > > > > > states
> > > > > > > > > > > > that
> > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > configurations
> > > > > > a
> > > > > > > > > > > connector
> > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > connector
> > > > > > > > > > > documentation
> > > > > > > > > > > > or
> > > > > > > > > > > > > > in the worst case, look directly at the connector
> > > > source
> > > > > > > > code.",
> > > > > > > > > > and
> > > > > > > > > > > > that
> > > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > > required
> > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > > "tools
> > > > > > will
> > > > > > > > be
> > > > > > > > > > > able
> > > > > > > > > > > > to
> > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > connectors".
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > > newly-proposed
> > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > allow users to do that they will not already be able
> > > > to do
> > > > > > > > with the
> > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > configuration
> > > > > > > > > > definitions
> > > > > > > > > > > in
> > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please take a look and let me know if you have any
> > > > > > feedback.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
Re: KIP-769: Connect API to retrieve connector configuration definitions
> > filtering the response to just grab the "definition" field of
> > > each
> > > > > > > element
> > > > > > > > > in the "configs" array in the response.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > viktorsomo...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Folks,
> > > > > > > > > >
> > > > > > > > > > I too think this would be a very useful feature. Some of our
> > > > > > > management
> > > > > > > > > > applications would provide a wizard for creating connectors.
> > > In
> > > > > this
> > > > > > > > > > scenario the user basically would fill out a sample
> > > configuration
> > > > > > > > > generated
> > > > > > > > > > by the UI which would send it back to Connect for validation
> > > and
> > > > > > > > > eventually
> > > > > > > > > > create a new connector. The first part of this workflow can
> > > be
> > > > > > > enhanced
> > > > > > > > > if
> > > > > > > > > > we had an API that can return the configuration definition
> > > of the
> > > > > > > given
> > > > > > > > > > type of connector as the UI application would be able to
> > > > > generate a
> > > > > > > > > sample
> > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > essentially
> > > > > > > > > works
> > > > > > > > > > and returns the data that we need, however it is a HTTP PUT
> > > API
> > > > > that
> > > > > > > is a
> > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > functionally
> > > > > > > > > > different as it validates the given (dummy) request. In case
> > > of
> > > > > sink
> > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > >
> > > > > > > > > > A suggestion for the KIP: I think it can be useful to return
> > > the
> > > > > > > config
> > > > > > > > > > groups and the connector class' name similarly to the
> > > validate
> > > > > API
> > > > > > > just
> > > > > > > > > in
> > > > > > > > > > case any frontend needs them (and also the response would be
> > > more
> > > > > > > like
> > > > > > > > > the
> > > > > > > > > > validate API but simpler).
> > > > > > > > > >
> > > > > > > > > > Viktor
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > ryannedo...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> > > the
> > > > > same
> > > > > > > > > > handler
> > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > mickael.mai...@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > You're right, you can achieve the same functionality
> > > using
> > > > > the
> > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > In my mind it was only for validation once you have
> > > build a
> > > > > > > > > > > > configuration but when used with an empty configuration,
> > > it
> > > > > > > basically
> > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > don't
> > > > > > > think it
> > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > > > section
> > > > > > > > > states
> > > > > > > > > > > that
> > > > > > > > > > > > > "Connect does not provide a way to see what
> > > configurations
> > > > > a
> > > > > > > > > > connector
> > > > > > > > > > > > > requires. Instead users have to go look at the
> > > connector
> > > > > > > > > > documentation
> > > > > > > > > > > or
> > > > > > > > > > > > > in the worst case, look directly at the connector
> > > source
> > > > > > > code.",
> > > > > > > > > and
> > > > > > > > > > > that
> > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > required
> > > > > > > > > > > > configurations
> > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > "tools
> > > > > will
> > > > > > > be
> > > > > > > > > > able
> > > > > > > > > > > to
> > > > > > > > > > > > > generate wizards for configuring and starting
> > > connectors".
> > > > > > > > > > > > >
> > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > newly-proposed
> > > > > > > > > > > endpoint
> > > > > > > > > > > > > allow users to do that they will not already be able
> > > to do
> > > > > > > with the
> > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > configuration
> > > > > > > > > definitions
> > > > > > > > > > in
> > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please take a look and let me know if you have any
> > > > > feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
Re: KIP-769: Connect API to retrieve connector configuration definitions
e that it'd be easier for
> people
> > > > > designing
> > > > > > > UIs
> > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > worth the
> > > > > > > > > additional implementation, testing, and maintenance
> burden. If
> > > it
> > > > > were
> > > > > > > > > possible to issue a PUT request without unexpected 500s for
> > > invalid
> > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > simple as
> > > > > > > issuing
> > > > > > > > > a PUT request with a dummy body consisting of nothing
> except
> > > the
> > > > > > > connector
> > > > > > > > > class (which at this point we might even make unnecessary
> and
> > > just
> > > > > > > > > automatically replace with the connector class from the
> URL)
> > > and
> > > > > then
> > > > > > > > > filtering the response to just grab the "definition" field
> of
> > > each
> > > > > > > element
> > > > > > > > > in the "configs" array in the response.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > viktorsomo...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Folks,
> > > > > > > > > >
> > > > > > > > > > I too think this would be a very useful feature. Some of
> our
> > > > > > > management
> > > > > > > > > > applications would provide a wizard for creating
> connectors.
> > > In
> > > > > this
> > > > > > > > > > scenario the user basically would fill out a sample
> > > configuration
> > > > > > > > > generated
> > > > > > > > > > by the UI which would send it back to Connect for
> validation
> > > and
> > > > > > > > > eventually
> > > > > > > > > > create a new connector. The first part of this workflow
> can
> > > be
> > > > > > > enhanced
> > > > > > > > > if
> > > > > > > > > > we had an API that can return the configuration
> definition
> > > of the
> > > > > > > given
> > > > > > > > > > type of connector as the UI application would be able to
> > > > > generate a
> > > > > > > > > sample
> > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > essentially
> > > > > > > > > works
> > > > > > > > > > and returns the data that we need, however it is a HTTP
> PUT
> > > API
> > > > > that
> > > > > > > is a
> > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > functionally
> > > > > > > > > > different as it validates the given (dummy) request. In
> case
> > > of
> > > > > sink
> > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > >
> > > > > > > > > > A suggestion for the KIP: I think it can be useful to
> return
> > > the
> > > > > > > config
> > > > > > > > > > groups and the connector class' name similarly to the
> > > validate
> > > > > API
> > > > > > > just
> > > > > > > > > in
> > > > > > > > > > case any frontend needs them (and also the response
> would be
> > > more
> > > > > > > like
> > > > > > > > > the
> > > > > > > > > > validate API but simpler).
> > > > > > > > > >
> > > > > > > > > > Viktor
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > ryannedo...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I think it'd be worth adding a GET version, fwiw.
> Could be
> > > the
> > > > > same
> > > > > > > > > > handler
> > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > mickael.mai...@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > You're right, you can achieve the same functionality
> > > using
> > > > > the
> > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > In my mind it was only for validation once you have
> > > build a
> > > > > > > > > > > > configuration but when used with an empty
> configuration,
> > > it
> > > > > > > basically
> > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > don't
> > > > > > > think it
> > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm wondering about the use case here. The
> motivation
> > > > > section
> > > > > > > > > states
> > > > > > > > > > > that
> > > > > > > > > > > > > "Connect does not provide a way to see what
> > > configurations
> > > > > a
> > > > > > > > > > connector
> > > > > > > > > > > > > requires. Instead users have to go look at the
> > > connector
> > > > > > > > > > documentation
> > > > > > > > > > > or
> > > > > > > > > > > > > in the worst case, look directly at the connector
> > > source
> > > > > > > code.",
> > > > > > > > > and
> > > > > > > > > > > that
> > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > required
> > > > > > > > > > > > configurations
> > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > "tools
> > > > > will
> > > > > > > be
> > > > > > > > > > able
> > > > > > > > > > > to
> > > > > > > > > > > > > generate wizards for configuring and starting
> > > connectors".
> > > > > > > > > > > > >
> > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > newly-proposed
> > > > > > > > > > > endpoint
> > > > > > > > > > > > > allow users to do that they will not already be
> able
> > > to do
> > > > > > > with the
> > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > configuration
> > > > > > > > > definitions
> > > > > > > > > > in
> > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please take a look and let me know if you have
> any
> > > > > feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
Re: KIP-769: Connect API to retrieve connector configuration definitions
er basically would fill out a sample
> > configuration
> > > > > > > > generated
> > > > > > > > > by the UI which would send it back to Connect for validation
> > and
> > > > > > > > eventually
> > > > > > > > > create a new connector. The first part of this workflow can
> > be
> > > > > > enhanced
> > > > > > > > if
> > > > > > > > > we had an API that can return the configuration definition
> > of the
> > > > > > given
> > > > > > > > > type of connector as the UI application would be able to
> > > > generate a
> > > > > > > > sample
> > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > essentially
> > > > > > > > works
> > > > > > > > > and returns the data that we need, however it is a HTTP PUT
> > API
> > > > that
> > > > > > is a
> > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > functionally
> > > > > > > > > different as it validates the given (dummy) request. In case
> > of
> > > > sink
> > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > >
> > > > > > > > > A suggestion for the KIP: I think it can be useful to return
> > the
> > > > > > config
> > > > > > > > > groups and the connector class' name similarly to the
> > validate
> > > > API
> > > > > > just
> > > > > > > > in
> > > > > > > > > case any frontend needs them (and also the response would be
> > more
> > > > > > like
> > > > > > > > the
> > > > > > > > > validate API but simpler).
> > > > > > > > >
> > > > > > > > > Viktor
> > > > > > > > >
> > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > ryannedo...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> > the
> > > > same
> > > > > > > > > handler
> > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > mickael.mai...@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Chris,
> > > > > > > > > > >
> > > > > > > > > > > You're right, you can achieve the same functionality
> > using
> > > > the
> > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > In my mind it was only for validation once you have
> > build a
> > > > > > > > > > > configuration but when used with an empty configuration,
> > it
> > > > > > basically
> > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > >
> > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > don't
> > > > > > think it
> > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > > section
> > > > > > > > states
> > > > > > > > > > that
> > > > > > > > > > > > "Connect does not provide a way to see what
> > configurations
> > > > a
> > > > > > > > > connector
> > > > > > > > > > > > requires. Instead users have to go look at the
> > connector
> > > > > > > > > documentation
> > > > > > > > > > or
> > > > > > > > > > > > in the worst case, look directly at the connector
> > source
> > > > > > code.",
> > > > > > > > and
> > > > > > > > > > that
> > > > > > > > > > > > with this KIP, "users will be able to discover the
> > required
> > > > > > > > > > > configurations
> > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > "tools
> > > > will
> > > > > > be
> > > > > > > > > able
> > > > > > > > > > to
> > > > > > > > > > > > generate wizards for configuring and starting
> > connectors".
> > > > > > > > > > > >
> > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > endpoint not address these points? What will the
> > > > newly-proposed
> > > > > > > > > > endpoint
> > > > > > > > > > > > allow users to do that they will not already be able
> > to do
> > > > > > with the
> > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've created KIP-769 to expose connector
> > configuration
> > > > > > > > definitions
> > > > > > > > > in
> > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please take a look and let me know if you have any
> > > > feedback.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
Re: KIP-769: Connect API to retrieve connector configuration definitions
> > > > With regards to 2, this is technically possible but I'm unsure
> > > it'd be
> > > > > too
> > > > > > > common out in the wild given that most validations that could
> be
> > > > > expensive
> > > > > > > would involve things like connecting to a database, checking
> if a
> > > cloud
> > > > > > > storage bucket exists, etc., none of which are possible without
> > > some
> > > > > > > configuration properties from the user (db hostname, bucket
> name,
> > > > > etc.).
> > > > > > >
> > > > > > > With regards to 3, I do agree that it'd be easier for people
> > > designing
> > > > > UIs
> > > > > > > to have a GET API to work against. I'm just not sure it's
> worth the
> > > > > > > additional implementation, testing, and maintenance burden. If
> it
> > > were
> > > > > > > possible to issue a PUT request without unexpected 500s for
> invalid
> > > > > > > configs, would that suffice? AFAICT it'd basically be as
> simple as
> > > > > issuing
> > > > > > > a PUT request with a dummy body consisting of nothing except
> the
> > > > > connector
> > > > > > > class (which at this point we might even make unnecessary and
> just
> > > > > > > automatically replace with the connector class from the URL)
> and
> > > then
> > > > > > > filtering the response to just grab the "definition" field of
> each
> > > > > element
> > > > > > > in the "configs" array in the response.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > viktorsomo...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Folks,
> > > > > > > >
> > > > > > > > I too think this would be a very useful feature. Some of our
> > > > > management
> > > > > > > > applications would provide a wizard for creating connectors.
> In
> > > this
> > > > > > > > scenario the user basically would fill out a sample
> configuration
> > > > > > > generated
> > > > > > > > by the UI which would send it back to Connect for validation
> and
> > > > > > > eventually
> > > > > > > > create a new connector. The first part of this workflow can
> be
> > > > > enhanced
> > > > > > > if
> > > > > > > > we had an API that can return the configuration definition
> of the
> > > > > given
> > > > > > > > type of connector as the UI application would be able to
> > > generate a
> > > > > > > sample
> > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > essentially
> > > > > > > works
> > > > > > > > and returns the data that we need, however it is a HTTP PUT
> API
> > > that
> > > > > is a
> > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > functionally
> > > > > > > > different as it validates the given (dummy) request. In case
> of
> > > sink
> > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > >
> > > > > > > > A suggestion for the KIP: I think it can be useful to return
> the
> > > > > config
> > > > > > > > groups and the connector class' name similarly to the
> validate
> > > API
> > > > > just
> > > > > > > in
> > > > > > > > case any frontend needs them (and also the response would be
> more
> > > > > like
> > > > > > > the
> > > > > > > > validate API but simpler).
> > > > > > > >
> > > > > > > > Viktor
> > > > > > > >
> > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > ryannedo...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> the
> > > same
> > > > > > > > handler
> > > > > > > > > with just a different spelling maybe.
> > > > > > > > >
> > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > mickael.mai...@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Chris,
> > > > > > > > > >
> > > > > > > > > > You're right, you can achieve the same functionality
> using
> > > the
> > > > > > > > > > existing validate endpoint.
> > > > > > > > > > In my mind it was only for validation once you have
> build a
> > > > > > > > > > configuration but when used with an empty configuration,
> it
> > > > > basically
> > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > >
> > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> don't
> > > > > think it
> > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Mickael,
> > > > > > > > > > >
> > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > section
> > > > > > > states
> > > > > > > > > that
> > > > > > > > > > > "Connect does not provide a way to see what
> configurations
> > > a
> > > > > > > > connector
> > > > > > > > > > > requires. Instead users have to go look at the
> connector
> > > > > > > > documentation
> > > > > > > > > or
> > > > > > > > > > > in the worst case, look directly at the connector
> source
> > > > > code.",
> > > > > > > and
> > > > > > > > > that
> > > > > > > > > > > with this KIP, "users will be able to discover the
> required
> > > > > > > > > > configurations
> > > > > > > > > > > for connectors installed in a Connect cluster" and
> "tools
> > > will
> > > > > be
> > > > > > > > able
> > > > > > > > > to
> > > > > > > > > > > generate wizards for configuring and starting
> connectors".
> > > > > > > > > > >
> > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > endpoint not address these points? What will the
> > > newly-proposed
> > > > > > > > > endpoint
> > > > > > > > > > > allow users to do that they will not already be able
> to do
> > > > > with the
> > > > > > > > > > > existing endpoint?
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I've created KIP-769 to expose connector
> configuration
> > > > > > > definitions
> > > > > > > > in
> > > > > > > > > > > > the Connect API
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > >
> > > > > > > > > > > > Please take a look and let me know if you have any
> > > feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
>
Re: KIP-769: Connect API to retrieve connector configuration definitions
e user based on that (nicely drawn diagram:
> > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > The connector-plugins/{connectorType}/config/validate API
> > essentially
> > > > > > works
> > > > > > > and returns the data that we need, however it is a HTTP PUT API
> > that
> > > > is a
> > > > > > > bit unintuitive for a fetch-like functionality and also
> > functionally
> > > > > > > different as it validates the given (dummy) request. In case of
> > sink
> > > > > > > connectors one would need to also provide a topic name.
> > > > > > >
> > > > > > > A suggestion for the KIP: I think it can be useful to return the
> > > > config
> > > > > > > groups and the connector class' name similarly to the validate
> > API
> > > > just
> > > > > > in
> > > > > > > case any frontend needs them (and also the response would be more
> > > > like
> > > > > > the
> > > > > > > validate API but simpler).
> > > > > > >
> > > > > > > Viktor
> > > > > > >
> > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > ryannedo...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be the
> > same
> > > > > > > handler
> > > > > > > > with just a different spelling maybe.
> > > > > > > >
> > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > mickael.mai...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Chris,
> > > > > > > > >
> > > > > > > > > You're right, you can achieve the same functionality using
> > the
> > > > > > > > > existing validate endpoint.
> > > > > > > > > In my mind it was only for validation once you have build a
> > > > > > > > > configuration but when used with an empty configuration, it
> > > > basically
> > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > >
> > > > > > > > > I think it's a bit easier to use a GET endpoint but I don't
> > > > think it
> > > > > > > > > really warrants a different endpoint.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Mickael,
> > > > > > > > > >
> > > > > > > > > > I'm wondering about the use case here. The motivation
> > section
> > > > > > states
> > > > > > > > that
> > > > > > > > > > "Connect does not provide a way to see what configurations
> > a
> > > > > > > connector
> > > > > > > > > > requires. Instead users have to go look at the connector
> > > > > > > documentation
> > > > > > > > or
> > > > > > > > > > in the worst case, look directly at the connector source
> > > > code.",
> > > > > > and
> > > > > > > > that
> > > > > > > > > > with this KIP, "users will be able to discover the required
> > > > > > > > > configurations
> > > > > > > > > > for connectors installed in a Connect cluster" and "tools
> > will
> > > > be
> > > > > > > able
> > > > > > > > to
> > > > > > > > > > generate wizards for configuring and starting connectors".
> > > > > > > > > >
> > > > > > > > > > Does the existing "PUT
> > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > endpoint not address these points? What will the
> > newly-proposed
> > > > > > > > endpoint
> > > > > > > > > > allow users to do that they will not already be able to do
> > > > with the
> > > > > > > > > > existing endpoint?
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I've created KIP-769 to expose connector configuration
> > > > > > definitions
> > > > > > > in
> > > > > > > > > > > the Connect API
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > >
> > > > > > > > > > > Please take a look and let me know if you have any
> > feedback.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
Re: KIP-769: Connect API to retrieve connector configuration definitions
y possible but I'm unsure
> it'd be
> > > too
> > > > > common out in the wild given that most validations that could be
> > > expensive
> > > > > would involve things like connecting to a database, checking if a
> cloud
> > > > > storage bucket exists, etc., none of which are possible without
> some
> > > > > configuration properties from the user (db hostname, bucket name,
> > > etc.).
> > > > >
> > > > > With regards to 3, I do agree that it'd be easier for people
> designing
> > > UIs
> > > > > to have a GET API to work against. I'm just not sure it's worth the
> > > > > additional implementation, testing, and maintenance burden. If it
> were
> > > > > possible to issue a PUT request without unexpected 500s for invalid
> > > > > configs, would that suffice? AFAICT it'd basically be as simple as
> > > issuing
> > > > > a PUT request with a dummy body consisting of nothing except the
> > > connector
> > > > > class (which at this point we might even make unnecessary and just
> > > > > automatically replace with the connector class from the URL) and
> then
> > > > > filtering the response to just grab the "definition" field of each
> > > element
> > > > > in the "configs" array in the response.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > viktorsomo...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Folks,
> > > > > >
> > > > > > I too think this would be a very useful feature. Some of our
> > > management
> > > > > > applications would provide a wizard for creating connectors. In
> this
> > > > > > scenario the user basically would fill out a sample configuration
> > > > > generated
> > > > > > by the UI which would send it back to Connect for validation and
> > > > > eventually
> > > > > > create a new connector. The first part of this workflow can be
> > > enhanced
> > > > > if
> > > > > > we had an API that can return the configuration definition of the
> > > given
> > > > > > type of connector as the UI application would be able to
> generate a
> > > > > sample
> > > > > > for the user based on that (nicely drawn diagram:
> > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > The connector-plugins/{connectorType}/config/validate API
> essentially
> > > > > works
> > > > > > and returns the data that we need, however it is a HTTP PUT API
> that
> > > is a
> > > > > > bit unintuitive for a fetch-like functionality and also
> functionally
> > > > > > different as it validates the given (dummy) request. In case of
> sink
> > > > > > connectors one would need to also provide a topic name.
> > > > > >
> > > > > > A suggestion for the KIP: I think it can be useful to return the
> > > config
> > > > > > groups and the connector class' name similarly to the validate
> API
> > > just
> > > > > in
> > > > > > case any frontend needs them (and also the response would be more
> > > like
> > > > > the
> > > > > > validate API but simpler).
> > > > > >
> > > > > > Viktor
> > > > > >
> > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> ryannedo...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I think it'd be worth adding a GET version, fwiw. Could be the
> same
> > > > > > handler
> > > > > > > with just a different spelling maybe.
> > > > > > >
> > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > mickael.mai...@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Chris,
> > > > > > > >
> > > > > > > > You're right, you can achieve the same functionality using
> the
> > > > > > > > existing validate endpoint.
> > > > > > > > In my mind it was only for validation once you have build a
> > > > > > > > configuration but when used with an empty configuration, it
> > > basically
> > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > >
> > > > > > > > I think it's a bit easier to use a GET endpoint but I don't
> > > think it
> > > > > > > > really warrants a different endpoint.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > I'm wondering about the use case here. The motivation
> section
> > > > > states
> > > > > > > that
> > > > > > > > > "Connect does not provide a way to see what configurations
> a
> > > > > > connector
> > > > > > > > > requires. Instead users have to go look at the connector
> > > > > > documentation
> > > > > > > or
> > > > > > > > > in the worst case, look directly at the connector source
> > > code.",
> > > > > and
> > > > > > > that
> > > > > > > > > with this KIP, "users will be able to discover the required
> > > > > > > > configurations
> > > > > > > > > for connectors installed in a Connect cluster" and "tools
> will
> > > be
> > > > > > able
> > > > > > > to
> > > > > > > > > generate wizards for configuring and starting connectors".
> > > > > > > > >
> > > > > > > > > Does the existing "PUT
> > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > endpoint not address these points? What will the
> newly-proposed
> > > > > > > endpoint
> > > > > > > > > allow users to do that they will not already be able to do
> > > with the
> > > > > > > > > existing endpoint?
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > mickael.mai...@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I've created KIP-769 to expose connector configuration
> > > > > definitions
> > > > > > in
> > > > > > > > > > the Connect API
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > >
> > > > > > > > > > Please take a look and let me know if you have any
> feedback.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
>
Re: KIP-769: Connect API to retrieve connector configuration definitions
ck to Connect for validation and
> > > > eventually
> > > > > create a new connector. The first part of this workflow can be
> > enhanced
> > > > if
> > > > > we had an API that can return the configuration definition of the
> > given
> > > > > type of connector as the UI application would be able to generate a
> > > > sample
> > > > > for the user based on that (nicely drawn diagram:
> > > > > https://imgur.com/a/7S1Xwm5).
> > > > > The connector-plugins/{connectorType}/config/validate API essentially
> > > > works
> > > > > and returns the data that we need, however it is a HTTP PUT API that
> > is a
> > > > > bit unintuitive for a fetch-like functionality and also functionally
> > > > > different as it validates the given (dummy) request. In case of sink
> > > > > connectors one would need to also provide a topic name.
> > > > >
> > > > > A suggestion for the KIP: I think it can be useful to return the
> > config
> > > > > groups and the connector class' name similarly to the validate API
> > just
> > > > in
> > > > > case any frontend needs them (and also the response would be more
> > like
> > > > the
> > > > > validate API but simpler).
> > > > >
> > > > > Viktor
> > > > >
> > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan
> > > > > wrote:
> > > > >
> > > > > > I think it'd be worth adding a GET version, fwiw. Could be the same
> > > > > handler
> > > > > > with just a different spelling maybe.
> > > > > >
> > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > mickael.mai...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > You're right, you can achieve the same functionality using the
> > > > > > > existing validate endpoint.
> > > > > > > In my mind it was only for validation once you have build a
> > > > > > > configuration but when used with an empty configuration, it
> > basically
> > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > >
> > > > > > > I think it's a bit easier to use a GET endpoint but I don't
> > think it
> > > > > > > really warrants a different endpoint.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > I'm wondering about the use case here. The motivation section
> > > > states
> > > > > > that
> > > > > > > > "Connect does not provide a way to see what configurations a
> > > > > connector
> > > > > > > > requires. Instead users have to go look at the connector
> > > > > documentation
> > > > > > or
> > > > > > > > in the worst case, look directly at the connector source
> > code.",
> > > > and
> > > > > > that
> > > > > > > > with this KIP, "users will be able to discover the required
> > > > > > > configurations
> > > > > > > > for connectors installed in a Connect cluster" and "tools will
> > be
> > > > > able
> > > > > > to
> > > > > > > > generate wizards for configuring and starting connectors".
> > > > > > > >
> > > > > > > > Does the existing "PUT
> > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > endpoint not address these points? What will the newly-proposed
> > > > > > endpoint
> > > > > > > > allow users to do that they will not already be able to do
> > with the
> > > > > > > > existing endpoint?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > mickael.mai...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I've created KIP-769 to expose connector configuration
> > > > definitions
> > > > > in
> > > > > > > > > the Connect API
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > >
> > > > > > > > > Please take a look and let me know if you have any feedback.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
Re: KIP-769: Connect API to retrieve connector configuration definitions
com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > You're right, you can achieve the same functionality using the
> > > > > > existing validate endpoint.
> > > > > > In my mind it was only for validation once you have build a
> > > > > > configuration but when used with an empty configuration, it
> basically
> > > > > > serves the same purpose as the proposed new endpoint.
> > > > > >
> > > > > > I think it's a bit easier to use a GET endpoint but I don't
> think it
> > > > > > really warrants a different endpoint.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > I'm wondering about the use case here. The motivation section
> > > states
> > > > > that
> > > > > > > "Connect does not provide a way to see what configurations a
> > > > connector
> > > > > > > requires. Instead users have to go look at the connector
> > > > documentation
> > > > > or
> > > > > > > in the worst case, look directly at the connector source
> code.",
> > > and
> > > > > that
> > > > > > > with this KIP, "users will be able to discover the required
> > > > > > configurations
> > > > > > > for connectors installed in a Connect cluster" and "tools will
> be
> > > > able
> > > > > to
> > > > > > > generate wizards for configuring and starting connectors".
> > > > > > >
> > > > > > > Does the existing "PUT
> > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > endpoint not address these points? What will the newly-proposed
> > > > > endpoint
> > > > > > > allow users to do that they will not already be able to do
> with the
> > > > > > > existing endpoint?
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > mickael.mai...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I've created KIP-769 to expose connector configuration
> > > definitions
> > > > in
> > > > > > > > the Connect API
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > >
> > > > > > > > Please take a look and let me know if you have any feedback.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: KIP-769: Connect API to retrieve connector configuration definitions
; sample
> > > for the user based on that (nicely drawn diagram:
> > > https://imgur.com/a/7S1Xwm5).
> > > The connector-plugins/{connectorType}/config/validate API essentially
> > works
> > > and returns the data that we need, however it is a HTTP PUT API that is a
> > > bit unintuitive for a fetch-like functionality and also functionally
> > > different as it validates the given (dummy) request. In case of sink
> > > connectors one would need to also provide a topic name.
> > >
> > > A suggestion for the KIP: I think it can be useful to return the config
> > > groups and the connector class' name similarly to the validate API just
> > in
> > > case any frontend needs them (and also the response would be more like
> > the
> > > validate API but simpler).
> > >
> > > Viktor
> > >
> > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan
> > > wrote:
> > >
> > > > I think it'd be worth adding a GET version, fwiw. Could be the same
> > > handler
> > > > with just a different spelling maybe.
> > > >
> > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison > >
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > You're right, you can achieve the same functionality using the
> > > > > existing validate endpoint.
> > > > > In my mind it was only for validation once you have build a
> > > > > configuration but when used with an empty configuration, it basically
> > > > > serves the same purpose as the proposed new endpoint.
> > > > >
> > > > > I think it's a bit easier to use a GET endpoint but I don't think it
> > > > > really warrants a different endpoint.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I'm wondering about the use case here. The motivation section
> > states
> > > > that
> > > > > > "Connect does not provide a way to see what configurations a
> > > connector
> > > > > > requires. Instead users have to go look at the connector
> > > documentation
> > > > or
> > > > > > in the worst case, look directly at the connector source code.",
> > and
> > > > that
> > > > > > with this KIP, "users will be able to discover the required
> > > > > configurations
> > > > > > for connectors installed in a Connect cluster" and "tools will be
> > > able
> > > > to
> > > > > > generate wizards for configuring and starting connectors".
> > > > > >
> > > > > > Does the existing "PUT
> > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > endpoint not address these points? What will the newly-proposed
> > > > endpoint
> > > > > > allow users to do that they will not already be able to do with the
> > > > > > existing endpoint?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > mickael.mai...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I've created KIP-769 to expose connector configuration
> > definitions
> > > in
> > > > > > > the Connect API
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > >
> > > > > > > Please take a look and let me know if you have any feedback.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > >
> > > >
> > >
> >
Re: KIP-769: Connect API to retrieve connector configuration definitions
> configuration but when used with an empty configuration, it basically
> > > > serves the same purpose as the proposed new endpoint.
> > > >
> > > > I think it's a bit easier to use a GET endpoint but I don't think it
> > > > really warrants a different endpoint.
> > > >
> > > > Thanks
> > > >
> > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I'm wondering about the use case here. The motivation section
> states
> > > that
> > > > > "Connect does not provide a way to see what configurations a
> > connector
> > > > > requires. Instead users have to go look at the connector
> > documentation
> > > or
> > > > > in the worst case, look directly at the connector source code.",
> and
> > > that
> > > > > with this KIP, "users will be able to discover the required
> > > > configurations
> > > > > for connectors installed in a Connect cluster" and "tools will be
> > able
> > > to
> > > > > generate wizards for configuring and starting connectors".
> > > > >
> > > > > Does the existing "PUT
> > > > /connector-plugins/{connector-type}/config/validate"
> > > > > endpoint not address these points? What will the newly-proposed
> > > endpoint
> > > > > allow users to do that they will not already be able to do with the
> > > > > existing endpoint?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > mickael.mai...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I've created KIP-769 to expose connector configuration
> definitions
> > in
> > > > > > the Connect API
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > >
> > > > > > Please take a look and let me know if you have any feedback.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > >
> > >
> >
>
Re: KIP-769: Connect API to retrieve connector configuration definitions
> > configurations
> > > > for connectors installed in a Connect cluster" and "tools will be
> able
> > to
> > > > generate wizards for configuring and starting connectors".
> > > >
> > > > Does the existing "PUT
> > > /connector-plugins/{connector-type}/config/validate"
> > > > endpoint not address these points? What will the newly-proposed
> > endpoint
> > > > allow users to do that they will not already be able to do with the
> > > > existing endpoint?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I've created KIP-769 to expose connector configuration definitions
> in
> > > > > the Connect API
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > >
> > > > > Please take a look and let me know if you have any feedback.
> > > > >
> > > > > Thanks
> > > > >
> > >
> >
>
Re: KIP-769: Connect API to retrieve connector configuration definitions
Hi Folks,
I too think this would be a very useful feature. Some of our management
applications would provide a wizard for creating connectors. In this
scenario the user basically would fill out a sample configuration generated
by the UI which would send it back to Connect for validation and eventually
create a new connector. The first part of this workflow can be enhanced if
we had an API that can return the configuration definition of the given
type of connector as the UI application would be able to generate a sample
for the user based on that (nicely drawn diagram:
https://imgur.com/a/7S1Xwm5).
The connector-plugins/{connectorType}/config/validate API essentially works
and returns the data that we need, however it is a HTTP PUT API that is a
bit unintuitive for a fetch-like functionality and also functionally
different as it validates the given (dummy) request. In case of sink
connectors one would need to also provide a topic name.
A suggestion for the KIP: I think it can be useful to return the config
groups and the connector class' name similarly to the validate API just in
case any frontend needs them (and also the response would be more like the
validate API but simpler).
Viktor
On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan wrote:
> I think it'd be worth adding a GET version, fwiw. Could be the same handler
> with just a different spelling maybe.
>
> On Fri, Aug 20, 2021, 7:44 AM Mickael Maison
> wrote:
>
> > Hi Chris,
> >
> > You're right, you can achieve the same functionality using the
> > existing validate endpoint.
> > In my mind it was only for validation once you have build a
> > configuration but when used with an empty configuration, it basically
> > serves the same purpose as the proposed new endpoint.
> >
> > I think it's a bit easier to use a GET endpoint but I don't think it
> > really warrants a different endpoint.
> >
> > Thanks
> >
> > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > wrote:
> > >
> > > Hi Mickael,
> > >
> > > I'm wondering about the use case here. The motivation section states
> that
> > > "Connect does not provide a way to see what configurations a connector
> > > requires. Instead users have to go look at the connector documentation
> or
> > > in the worst case, look directly at the connector source code.", and
> that
> > > with this KIP, "users will be able to discover the required
> > configurations
> > > for connectors installed in a Connect cluster" and "tools will be able
> to
> > > generate wizards for configuring and starting connectors".
> > >
> > > Does the existing "PUT
> > /connector-plugins/{connector-type}/config/validate"
> > > endpoint not address these points? What will the newly-proposed
> endpoint
> > > allow users to do that they will not already be able to do with the
> > > existing endpoint?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I've created KIP-769 to expose connector configuration definitions in
> > > > the Connect API
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > >
> > > > Please take a look and let me know if you have any feedback.
> > > >
> > > > Thanks
> > > >
> >
>
Re: KIP-769: Connect API to retrieve connector configuration definitions
I think it'd be worth adding a GET version, fwiw. Could be the same handler
with just a different spelling maybe.
On Fri, Aug 20, 2021, 7:44 AM Mickael Maison
wrote:
> Hi Chris,
>
> You're right, you can achieve the same functionality using the
> existing validate endpoint.
> In my mind it was only for validation once you have build a
> configuration but when used with an empty configuration, it basically
> serves the same purpose as the proposed new endpoint.
>
> I think it's a bit easier to use a GET endpoint but I don't think it
> really warrants a different endpoint.
>
> Thanks
>
> On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> wrote:
> >
> > Hi Mickael,
> >
> > I'm wondering about the use case here. The motivation section states that
> > "Connect does not provide a way to see what configurations a connector
> > requires. Instead users have to go look at the connector documentation or
> > in the worst case, look directly at the connector source code.", and that
> > with this KIP, "users will be able to discover the required
> configurations
> > for connectors installed in a Connect cluster" and "tools will be able to
> > generate wizards for configuring and starting connectors".
> >
> > Does the existing "PUT
> /connector-plugins/{connector-type}/config/validate"
> > endpoint not address these points? What will the newly-proposed endpoint
> > allow users to do that they will not already be able to do with the
> > existing endpoint?
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison >
> > wrote:
> >
> > > Hi,
> > >
> > > I've created KIP-769 to expose connector configuration definitions in
> > > the Connect API
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > >
> > > Please take a look and let me know if you have any feedback.
> > >
> > > Thanks
> > >
>
Re: KIP-769: Connect API to retrieve connector configuration definitions
Hi Chris,
You're right, you can achieve the same functionality using the
existing validate endpoint.
In my mind it was only for validation once you have build a
configuration but when used with an empty configuration, it basically
serves the same purpose as the proposed new endpoint.
I think it's a bit easier to use a GET endpoint but I don't think it
really warrants a different endpoint.
Thanks
On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
wrote:
>
> Hi Mickael,
>
> I'm wondering about the use case here. The motivation section states that
> "Connect does not provide a way to see what configurations a connector
> requires. Instead users have to go look at the connector documentation or
> in the worst case, look directly at the connector source code.", and that
> with this KIP, "users will be able to discover the required configurations
> for connectors installed in a Connect cluster" and "tools will be able to
> generate wizards for configuring and starting connectors".
>
> Does the existing "PUT /connector-plugins/{connector-type}/config/validate"
> endpoint not address these points? What will the newly-proposed endpoint
> allow users to do that they will not already be able to do with the
> existing endpoint?
>
> Cheers,
>
> Chris
>
> On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison
> wrote:
>
> > Hi,
> >
> > I've created KIP-769 to expose connector configuration definitions in
> > the Connect API
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> >
> > Please take a look and let me know if you have any feedback.
> >
> > Thanks
> >
Re: KIP-769: Connect API to retrieve connector configuration definitions
Hi Mickael,
I'm wondering about the use case here. The motivation section states that
"Connect does not provide a way to see what configurations a connector
requires. Instead users have to go look at the connector documentation or
in the worst case, look directly at the connector source code.", and that
with this KIP, "users will be able to discover the required configurations
for connectors installed in a Connect cluster" and "tools will be able to
generate wizards for configuring and starting connectors".
Does the existing "PUT /connector-plugins/{connector-type}/config/validate"
endpoint not address these points? What will the newly-proposed endpoint
allow users to do that they will not already be able to do with the
existing endpoint?
Cheers,
Chris
On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison
wrote:
> Hi,
>
> I've created KIP-769 to expose connector configuration definitions in
> the Connect API
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
>
> Please take a look and let me know if you have any feedback.
>
> Thanks
>
KIP-769: Connect API to retrieve connector configuration definitions
Hi, I've created KIP-769 to expose connector configuration definitions in the Connect API https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions Please take a look and let me know if you have any feedback. Thanks
