What do you think of "type" field remaining in, and returning
"transformation", "converter" or whatever type it is. This way the schema
remains consistent, and you can programmatically understand what plugins
are returned on the holistic "GET /plugins" endpoint? It will be slightly
redundant in the case where you specify the plugin-type as a path parameter.
On Thu, Apr 15, 2021 at 1:13 PM Tom Bentley <tbent...@redhat.com> wrote:
> Hi Cyrus,
>
> Re 2: A very minor thing but while type=source|sink for a connector, it
> doesn't makes sense for the other plugin types, but so the json for those
> plugins should omit that property rather than have type=null.
>
> Apart from that it seems reasonable to me. Thanks again,
>
> Tom
>
> On Thu, Apr 15, 2021 at 3:49 PM Cyrus Vafadari <cvafad...@gmail.com>
> wrote:
>
> > Hi Tom,
> >
> > Thanks for taking the time to respond with thoughtful questions:
> >
> > 1. I propose HTTP-400, will update the KIP to reflect this proposal
> > 2. I think the schema should be of the same format. It is quite minimal,
> so
> > there is room to add fields in the future without breaking compatibility.
> > I will update the KIP to specify.
> > 3. Yes, that's right.
> > 4. I personally don't see the value in deprecating since it's a simple
> > alias (interface and impl will both be simple changes). I would be
> > comfortable kicking that can down the road if/when there is a need for an
> > actual breaking change. That way we can keep the scope and diff here
> tight.
> > But iff the community feels strongly that deprecating is the right thing
> to
> > do, I'm happy to update the KIP to propose deprecating.
> >
> > Thanks again!
> >
> > On Wed, Apr 14, 2021 at 9:38 AM Tom Bentley <tbent...@redhat.com> wrote:
> >
> > > Hi Cyrus,
> > >
> > > Thanks for the KIP. A few questions:
> > >
> > > 1. What status code does the /plugins/{plugin-type} endpoint return
> when
> > an
> > > unknown type is given?
> > > 2. The result of /connector-plugins is a list of objects with 'class',
> > > 'type' and 'version' properties. Presumably /plugins/connector is the
> > same,
> > > but what is the schema for the other plugin types?
> > > 3. You're not proposing an equivalent of the
> > > /connector-plugins/{connectorType}/config/validate endpoint for
> > > non-connector types, which I think makes sense, because you would
> > validate
> > > an SMT's config via its used by a connector, so the existing endpoint
> > > suffices, right?
> > > 4. Would /connector-plugins eventually be deprecated in favour of
> > > /plugins/connector, or do we expect it to remain in the API
> indefinitely
> > > and accept that /connector-plugins and /plugins/connector provide
> > identical
> > > responses? If we had the intention to deprecate in the future maybe we
> > > should add a /plugins/connector/config/validate endpoint now?
> > >
> > > Many thanks,
> > >
> > > Tom
> > >
> > > On Tue, Apr 13, 2021 at 6:46 PM Cyrus Vafadari <cvafad...@gmail.com>
> > > wrote:
> > >
> > > > Hello all,
> > > >
> > > > As the number of connect plugins, SMT's, etc have grown, I wanted to
> > bump
> > > > this thread to see if there is more interest in adding a Connect REST
> > > > endpoint to get the current plugins in the worker.
> > > >
> > > > Thanks all, and thanks Chris for the initial feedback on this.
> > > >
> > > > On Mon, Feb 17, 2020 at 12:56 PM Cyrus Vafadari <cvafad...@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Thanks for the feedback, Chris.
> > > > >
> > > > > WRT Worker-plugins, I see your point that they aren't very useful
> for
> > > > > people trying to create connectors to see what exists already, but
> I
> > > > would
> > > > > find them useful for things like making assertions against a docker
> > > image
> > > > > to confirm the pluginpaths/classpaths are configured correctly. It
> is
> > > > more
> > > > > useful for "sanity check" kinds of things than exploring/browsing.
> > That
> > > > > said, I don't feel too strongly and if more feedback from the
> > community
> > > > > thinks they are better excluded then we can remove them.
> > > > >
> > > > > WRT camelCase vs hyphens, I see your point here, will update the
> KIP.
> > > > >
> > > > > WRT compatibility -- Good question -- I would expect that yes, new
> > > > plugins
> > > > > would match this format. I will add this explicitly in the KIP for
> > > > clarity.
> > > > > I do think it's a valuable, simple feature to have the worker able
> to
> > > > > report new plugins.
> > > > >
> > > > > Thanks again!
> > > > >
> > > > > On Wed, Feb 12, 2020 at 4:44 PM Christopher Egerton <
> > > chr...@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Cyrus,
> > > > >>
> > > > >> Thanks for the KIP!
> > > > >>
> > > > >> One quick question--I see the use case for being able to list
> > > > >> per-connector
> > > > >> plugins (SMTs, converters, and header converters) via REST API,
> but
> > > I'm
> > > > >> not
> > > > >> so sure about worker plugins (REST extensions and config
> providers).
> > > > Since
> > > > >> those are configured at startup for the worker, is there any
> > advantage
> > > > to
> > > > >> being able to see the worker plugins available on a running
> worker?
> > > It's
> > > > >> not like they can be added during the lifetime of that worker, and
> > > that
> > > > >> information wouldn't be useful to anyone trying to create a
> > connector
> > > as
> > > > >> they wouldn't be able to include worker plugins in a connector
> > config
> > > > >> anyways.
> > > > >>
> > > > >> And a small nit--it seems like the precedent set ever-so-slightly
> by
> > > the
> > > > >> /connector-plugins resource is to use a hyphen to separate a
> series
> > of
> > > > >> lowercase words in an endpoint path as opposed to camel case with
> > > > >> capitalization. What do you think about changing the possible
> plugin
> > > > types
> > > > >> to "connector", "converter", "header-converter", etc.?
> > > > >>
> > > > >> One final question about forwards compatibility--the description
> for
> > > the
> > > > >> /plugins endpoint indicates that it "Returns plugin descriptions
> of
> > > all
> > > > >> types". If another type of pluggable component is added to the
> > > > framework,
> > > > >> should we expect this endpoint to be updated to include that type
> of
> > > > >> plugin
> > > > >> as well? And if so, should we also expect a matching
> > > > /plugins/{pluginType}
> > > > >> endpoint to be added?
> > > > >>
> > > > >> Cheers,
> > > > >>
> > > > >> Chris
> > > > >>
> > > > >> On Fri, Sep 6, 2019 at 11:48 AM Cyrus Vafadari <
> cy...@confluent.io>
> > > > >> wrote:
> > > > >>
> > > > >> > I've updated the KIP here to simplify and clarify the goal --
> > it's a
> > > > >> pretty
> > > > >> > simple KIP to add a REST endpoint for SMTs and other plugins.
> This
> > > > will
> > > > >> > make it easier to see what SMTs you've loaded.
> > > > >> >
> > > > >> > Hoping for some good discussion!
> > > > >> >
> > > > >> > Thanks, all
> > > > >> >
> > > > >> > On Sat, Jul 20, 2019 at 9:07 PM Cyrus Vafadari <
> > cy...@confluent.io>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hello, all,
> > > > >> > >
> > > > >> > > I'd like to start discussion on a new KIP I'm proposing to add
> > > HTTP
> > > > >> > > endpoints to Kafka Connect to give us some more insight into
> > other
> > > > >> > > non-connector plugins, like Simple Message Transformations
> > (SMTs).
> > > > The
> > > > >> > KIP
> > > > >> > > would not change how Connect works in any meaningful way
> except
> > to
> > > > >> expose
> > > > >> > > some more data by the endpoint, analogous to existing
> > > > >> > > "ConnectorPluginsResource" class.
> > > > >> > >
> > > > >> > > Looking forward to getting some feedback.
> > > > >> > >
> > > > >> > > Cyrus
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-494%3A+Connect+REST+Endpoint+for+Transformations+%28SMTs%29+and+other+Plugins
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
