Hi,
I'm +1 for adding a GET endpoint for obtaining config definitions. It
always felt odd to me that one has to issue a PUT for that purpose. If
nothing else, it'd be better in terms of discoverability of the KC REST API.
One additional feature request I'd have is to expose the valid enum
constants for enum-typed options. That'll help to display the values in a
drop-down or via radio buttons in a UI, give us tab completion in kcctl,
etc.
Best,
--Gunnar
Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
<chr...@confluent.io.invalid>:
> Hi Viktor,
>
> It sounds like there are three major points here in favor of a new GET
> endpoint for connector config defs.
>
> 1. You cannot issue a blank ("dummy") request for sink connectors because a
> topic list/topic regex has to be supplied (otherwise the PUT endpoint
> returns a 500 response)
> 2. A dummy request still triggers custom validations by the connector,
> which may be best to avoid if we know for sure that the config isn't worth
> validating yet
> 3. It's more ergonomic and intuitive to be able to issue a GET request
> without having to give a dummy connector config
>
> With regards to 1, this is actually a bug in Connect (
> https://issues.apache.org/jira/browse/KAFKA-13327) with a fix already
> implemented and awaiting committer review (
> https://github.com/apache/kafka/pull/11369). I think it'd be better to
> focus on fixing this bug in general instead of implementing a new REST
> endpoint in order to allow people to work around it.
>
> With regards to 2, this is technically possible but I'm unsure it'd be too
> common out in the wild given that most validations that could be expensive
> would involve things like connecting to a database, checking if a cloud
> storage bucket exists, etc., none of which are possible without some
> configuration properties from the user (db hostname, bucket name, etc.).
>
> With regards to 3, I do agree that it'd be easier for people designing UIs
> to have a GET API to work against. I'm just not sure it's worth the
> additional implementation, testing, and maintenance burden. If it were
> possible to issue a PUT request without unexpected 500s for invalid
> configs, would that suffice? AFAICT it'd basically be as simple as issuing
> a PUT request with a dummy body consisting of nothing except the connector
> class (which at this point we might even make unnecessary and just
> automatically replace with the connector class from the URL) and then
> filtering the response to just grab the "definition" field of each element
> in the "configs" array in the response.
>
> Cheers,
>
> Chris
>
> On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> wrote:
>
> > Hi Folks,
> >
> > I too think this would be a very useful feature. Some of our management
> > applications would provide a wizard for creating connectors. In this
> > scenario the user basically would fill out a sample configuration
> generated
> > by the UI which would send it back to Connect for validation and
> eventually
> > create a new connector. The first part of this workflow can be enhanced
> if
> > we had an API that can return the configuration definition of the given
> > type of connector as the UI application would be able to generate a
> sample
> > for the user based on that (nicely drawn diagram:
> > https://imgur.com/a/7S1Xwm5).
> > The connector-plugins/{connectorType}/config/validate API essentially
> works
> > and returns the data that we need, however it is a HTTP PUT API that is a
> > bit unintuitive for a fetch-like functionality and also functionally
> > different as it validates the given (dummy) request. In case of sink
> > connectors one would need to also provide a topic name.
> >
> > A suggestion for the KIP: I think it can be useful to return the config
> > groups and the connector class' name similarly to the validate API just
> in
> > case any frontend needs them (and also the response would be more like
> the
> > validate API but simpler).
> >
> > Viktor
> >
> > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <ryannedo...@gmail.com>
> > wrote:
> >
> > > I think it'd be worth adding a GET version, fwiw. Could be the same
> > handler
> > > with just a different spelling maybe.
> > >
> > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <mickael.mai...@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > You're right, you can achieve the same functionality using the
> > > > existing validate endpoint.
> > > > In my mind it was only for validation once you have build a
> > > > configuration but when used with an empty configuration, it basically
> > > > serves the same purpose as the proposed new endpoint.
> > > >
> > > > I think it's a bit easier to use a GET endpoint but I don't think it
> > > > really warrants a different endpoint.
> > > >
> > > > Thanks
> > > >
> > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > <chr...@confluent.io.invalid> wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I'm wondering about the use case here. The motivation section
> states
> > > that
> > > > > "Connect does not provide a way to see what configurations a
> > connector
> > > > > requires. Instead users have to go look at the connector
> > documentation
> > > or
> > > > > in the worst case, look directly at the connector source code.",
> and
> > > that
> > > > > with this KIP, "users will be able to discover the required
> > > > configurations
> > > > > for connectors installed in a Connect cluster" and "tools will be
> > able
> > > to
> > > > > generate wizards for configuring and starting connectors".
> > > > >
> > > > > Does the existing "PUT
> > > > /connector-plugins/{connector-type}/config/validate"
> > > > > endpoint not address these points? What will the newly-proposed
> > > endpoint
> > > > > allow users to do that they will not already be able to do with the
> > > > > existing endpoint?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > mickael.mai...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I've created KIP-769 to expose connector configuration
> definitions
> > in
> > > > > > the Connect API
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > >
> > > > > > Please take a look and let me know if you have any feedback.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > >
> > >
> >
>
