Hi Viktor,
It sounds like there are three major points here in favor of a new GET
endpoint for connector config defs.
1. You cannot issue a blank ("dummy") request for sink connectors because a
topic list/topic regex has to be supplied (otherwise the PUT endpoint
returns a 500 response)
2. A dummy request still triggers custom validations by the connector,
which may be best to avoid if we know for sure that the config isn't worth
validating yet
3. It's more ergonomic and intuitive to be able to issue a GET request
without having to give a dummy connector config
With regards to 1, this is actually a bug in Connect (
https://issues.apache.org/jira/browse/KAFKA-13327) with a fix already
implemented and awaiting committer review (
https://github.com/apache/kafka/pull/11369). I think it'd be better to
focus on fixing this bug in general instead of implementing a new REST
endpoint in order to allow people to work around it.
With regards to 2, this is technically possible but I'm unsure it'd be too
common out in the wild given that most validations that could be expensive
would involve things like connecting to a database, checking if a cloud
storage bucket exists, etc., none of which are possible without some
configuration properties from the user (db hostname, bucket name, etc.).
With regards to 3, I do agree that it'd be easier for people designing UIs
to have a GET API to work against. I'm just not sure it's worth the
additional implementation, testing, and maintenance burden. If it were
possible to issue a PUT request without unexpected 500s for invalid
configs, would that suffice? AFAICT it'd basically be as simple as issuing
a PUT request with a dummy body consisting of nothing except the connector
class (which at this point we might even make unnecessary and just
automatically replace with the connector class from the URL) and then
filtering the response to just grab the "definition" field of each element
in the "configs" array in the response.
Cheers,
Chris
On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <viktorsomo...@gmail.com>
wrote:
> Hi Folks,
>
> I too think this would be a very useful feature. Some of our management
> applications would provide a wizard for creating connectors. In this
> scenario the user basically would fill out a sample configuration generated
> by the UI which would send it back to Connect for validation and eventually
> create a new connector. The first part of this workflow can be enhanced if
> we had an API that can return the configuration definition of the given
> type of connector as the UI application would be able to generate a sample
> for the user based on that (nicely drawn diagram:
> https://imgur.com/a/7S1Xwm5).
> The connector-plugins/{connectorType}/config/validate API essentially works
> and returns the data that we need, however it is a HTTP PUT API that is a
> bit unintuitive for a fetch-like functionality and also functionally
> different as it validates the given (dummy) request. In case of sink
> connectors one would need to also provide a topic name.
>
> A suggestion for the KIP: I think it can be useful to return the config
> groups and the connector class' name similarly to the validate API just in
> case any frontend needs them (and also the response would be more like the
> validate API but simpler).
>
> Viktor
>
> On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <ryannedo...@gmail.com>
> wrote:
>
> > I think it'd be worth adding a GET version, fwiw. Could be the same
> handler
> > with just a different spelling maybe.
> >
> > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <mickael.mai...@gmail.com>
> > wrote:
> >
> > > Hi Chris,
> > >
> > > You're right, you can achieve the same functionality using the
> > > existing validate endpoint.
> > > In my mind it was only for validation once you have build a
> > > configuration but when used with an empty configuration, it basically
> > > serves the same purpose as the proposed new endpoint.
> > >
> > > I think it's a bit easier to use a GET endpoint but I don't think it
> > > really warrants a different endpoint.
> > >
> > > Thanks
> > >
> > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > <chr...@confluent.io.invalid> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I'm wondering about the use case here. The motivation section states
> > that
> > > > "Connect does not provide a way to see what configurations a
> connector
> > > > requires. Instead users have to go look at the connector
> documentation
> > or
> > > > in the worst case, look directly at the connector source code.", and
> > that
> > > > with this KIP, "users will be able to discover the required
> > > configurations
> > > > for connectors installed in a Connect cluster" and "tools will be
> able
> > to
> > > > generate wizards for configuring and starting connectors".
> > > >
> > > > Does the existing "PUT
> > > /connector-plugins/{connector-type}/config/validate"
> > > > endpoint not address these points? What will the newly-proposed
> > endpoint
> > > > allow users to do that they will not already be able to do with the
> > > > existing endpoint?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I've created KIP-769 to expose connector configuration definitions
> in
> > > > > the Connect API
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > >
> > > > > Please take a look and let me know if you have any feedback.
> > > > >
> > > > > Thanks
> > > > >
> > >
> >
>
