Hi,
I have updated the KIP accordingly.
Thanks,
Mickael
On Fri, Jan 28, 2022 at 7:01 PM Chris Egerton
wrote:
>
> I think a simple approach here is fine. +1 for removing versioning for
> other plugins.
>
> On Fri, Jan 28, 2022 at 11:17 AM Mickael Maison
> wrote:
>
> > Thanks Chris and Tom for
I think a simple approach here is fine. +1 for removing versioning for
other plugins.
On Fri, Jan 28, 2022 at 11:17 AM Mickael Maison
wrote:
> Thanks Chris and Tom for the feedback.
>
> I'm having a hard time making a decision on this point!
>
> Re-reading your previous comments, I agree that
Thanks Chris and Tom for the feedback.
I'm having a hard time making a decision on this point!
Re-reading your previous comments, I agree that we're lacking a
contract on the version field, and we can see value (supporting
multiple versions for example) if this was improved.
But I'm starting to
To be honest I'm very sceptical that you can evolve (without breaking
compatibility) from a situation of having a large number of plugins which
have already implemented version() to return whatever they please, to a
situation where you can rely on it as the basis for some feature to support
Hi Mickael,
+1 for the warning at startup, it'd be great if we could get everyone to
start versioning all of their components. Although given the current state
of WARN-level logging in Connect (see
https://issues.apache.org/jira/browse/KAFKA-7509) there might be too much
noise for most people to
Hi Chris,
Thanks for taking another look!
That's a good point which I should have mentioned in the KIP. I think
we can have a PossiblyVersioned interface that extends Versioned and
has the default implementation. I'd like to also print a warning at
startup if a plugin does not override
Hi Tom,
Thanks for the feedback.
You're right we don't have any contracts around the version field.
Like Chris I've only used the current field to find the exact source
code of connectors. Even if it's a limited use, I think that's enough
to justify adding it to all plugins and making them
> > > header_converters and predicates. As we also
> > want to
> > > > > retrieve
> > > > > > > > > > > > > > > > configdef for these too, I think it makes
> > sense to
> > > > > introduce a
> > > > > for
> > > > > > > > > > > > > > > it too.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > &
> > > > > > > >
> > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining
> > > config
> > > > > > > > > > definitions. It
> > > >
; > > > > > > > > > > > etc.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > >
favor of a
> > > > > > > > new
> > > > > > > > > > GET
> > > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. You cannot issue a blank
t; > > > > > > > > > >
> > > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > > > connectors
> > > > > > > > > > > because a
> > > > > > > > > > > > > topi
; > > > > https://issues.apache.org/jira/browse/KAFKA-13327)
> > with a fix
> > > > > > > > already
> > > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think
> > it'd be
> > > > > > > > better to
> >
lidating yet
> > > > > > > > > > > > 3. It's more ergonomic and intuitive to be able to
> > > > > > > > > > > > issue a GET
> > > > > > > > request
> > > > > > > > > > &g
t; > > > > > > With regards to 2, this is technically possible but I'm
> > > > > > > > > > > unsure
> > > > > > > it'd be
> > > > > > > > > too
> > > > > > > > &g
; REST
> > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > >
> > > > > > > > > > > With regards to 2, this is technically possible but I'm
> > > > > > > > >
> > > expensive
> > > > > > > > > > > would involve things like connecting to a database,
> checking
> > > > > if a
> > > > > > > cloud
> > > > > > > > > > > storage bucket exists, etc., none of which are
> possible without
> > > > > > > some
> > > > >
> > designing
> > > > > > > > UIs
> > > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > > worth the
> > > > > > > > > > additional implementation, testing, and maintenance burden.
> > > > > > > > > > If
> > > > it
> > > > > > were
> > > &g
each
> > > > > > > element
> > > > > > > > > in the "configs" array in the response.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > >
> > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > viktorsomo...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Folks,
> > > > > > > > > >
> > > > > > > > > > I too think this would be a very useful feature. Some of
> our
> > > > > > > management
> > > > > > > > > > applications would provide a wizard for creating
> connectors.
> > > In
> > > > > this
> > > > > > > > > > scenario the user basically would fill out a sample
> > > configuration
> > > > > > > > > generated
> > > > > > > > > > by the UI which would send it back to Connect for
> validation
> > > and
> > > > > > > > > eventually
> > > > > > > > > > create a new connector. The first part of this workflow
> can
> > > be
> > > > > > > enhanced
> > > > > > > > > if
> > > > > > > > > > we had an API that can return the configuration
> definition
> > > of the
> > > > > > > given
> > > > > > > > > > type of connector as the UI application would be able to
> > > > > generate a
> > > > > > > > > sample
> > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > essentially
> > > > > > > > > works
> > > > > > > > > > and returns the data that we need, however it is a HTTP
> PUT
> > > API
> > > > > that
> > > > > > > is a
> > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > functionally
> > > > > > > > > > different as it validates the given (dummy) request. In
> case
> > > of
> > > > > sink
> > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > >
> > > > > > > > > > A suggestion for the KIP: I think it can be useful to
> return
> > > the
> > > > > > > config
> > > > > > > > > > groups and the connector class' name similarly to the
> > > validate
> > > > > API
> > > > > > > just
> > > > > > > > > in
> > > > > > > > > > case any frontend needs them (and also the response
> would be
> > > more
> > > > > > > like
> > > > > > > > > the
> > > > > > > > > > validate API but simpler).
> > > > > > > > > >
> > > > > > > > > > Viktor
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > ryannedo...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I think it'd be worth adding a GET version, fwiw.
> Could be
> > > the
> > > > > same
> > > > > > > > > > handler
> > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > mickael.mai...@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > You're right, you can achieve the same functionality
> > > using
> > > > > the
> > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > In my mind it was only for validation once you have
> > > build a
> > > > > > > > > > > > configuration but when used with an empty
> configuration,
> > > it
> > > > > > > basically
> > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > don't
> > > > > > > think it
> > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm wondering about the use case here. The
> motivation
> > > > > section
> > > > > > > > > states
> > > > > > > > > > > that
> > > > > > > > > > > > > "Connect does not provide a way to see what
> > > configurations
> > > > > a
> > > > > > > > > > connector
> > > > > > > > > > > > > requires. Instead users have to go look at the
> > > connector
> > > > > > > > > > documentation
> > > > > > > > > > > or
> > > > > > > > > > > > > in the worst case, look directly at the connector
> > > source
> > > > > > > code.",
> > > > > > > > > and
> > > > > > > > > > > that
> > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > required
> > > > > > > > > > > > configurations
> > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > "tools
> > > > > will
> > > > > > > be
> > > > > > > > > > able
> > > > > > > > > > > to
> > > > > > > > > > > > > generate wizards for configuring and starting
> > > connectors".
> > > > > > > > > > > > >
> > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > newly-proposed
> > > > > > > > > > > endpoint
> > > > > > > > > > > > > allow users to do that they will not already be
> able
> > > to do
> > > > > > > with the
> > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > configuration
> > > > > > > > > definitions
> > > > > > > > > > in
> > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please take a look and let me know if you have
> any
> > > > > feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
er basically would fill out a sample
> > configuration
> > > > > > > > generated
> > > > > > > > > by the UI which would send it back to Connect for validation
> > and
> > > > > > > > eventually
> > > > > > > > > create a new connector. The first part of this workflow can
> &
t; > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > essentially
> > > > > > > works
> > > > > &
> > > > > > > groups and the connector class' name similarly to the validate
> > API
> > > > just
> > > > > > in
> > > > > > > case any frontend needs them (and also the response would be more
> > > > like
> > > > > > the
&
be worth adding a GET version, fwiw. Could be the
> same
> > > > > > handler
> > > > > > > with just a different spelling maybe.
> > > > > > >
> > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > mickael.mai...@g
; > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > >
> > > > > > > I think it's a bit easier to use a GET endpoint but I don't
> > think it
> > > > > > > really warrants a different endpoint.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > I'm wondering about the use case here. The motivation section
> > > > states
> > > > > > that
> > > > > > > > "Connect does not provide a way to see what configurations a
> > > > > connector
> > > > > > > > requires. Instead users have to go look at the connector
> > > > > documentation
> > > > > > or
> > > > > > > > in the worst case, look directly at the connector source
> > code.",
> > > > and
> > > > > > that
> > > > > > > > with this KIP, "users will be able to discover the required
> > > > > > > configurations
> > > > > > > > for connectors installed in a Connect cluster" and "tools will
> > be
> > > > > able
> > > > > > to
> > > > > > > > generate wizards for configuring and starting connectors".
> > > > > > > >
> > > > > > > > Does the existing "PUT
> > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > endpoint not address these points? What will the newly-proposed
> > > > > > endpoint
> > > > > > > > allow users to do that they will not already be able to do
> > with the
> > > > > > > > existing endpoint?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > mickael.mai...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I've created KIP-769 to expose connector configuration
> > > > definitions
> > > > > in
> > > > > > > > > the Connect API
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > >
> > > > > > > > > Please take a look and let me know if you have any feedback.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
;
> > > > > > Thanks
> > > > > >
> > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > >
ations a
> > > connector
> > > > > > requires. Instead users have to go look at the connector
> > > documentation
> > > > or
> > > > > > in the worst case, look directly at the connector source code.",
> > and
> > >
and
> > > that
> > > > > with this KIP, "users will be able to discover the required
> > > > configurations
> > > > > for connectors installed in a Connect cluster" and "tools will be
> > able
> > > to
> >
points? What will the newly-proposed
> > endpoint
> > > > allow users to do that they will not already be able to do with the
> > > > existing endpoint?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I've created KIP-769 to expose connector configuration definitions
> in
> > > > > the Connect API
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > >
> > > > > Please take a look and let me know if you have any feedback.
> > > > >
> > > > > Thanks
> > > > >
> > >
> >
>
endpoint
> > > allow users to do that they will not already be able to do with the
> > > existing endpoint?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> mickael.mai
oints? What will the newly-proposed endpoint
> > allow users to do that they will not already be able to do with the
> > existing endpoint?
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison >
> > wrote:
connector configuration definitions in
> > the Connect API
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> >
> > Please take a look and let me know if you have any feedback.
> >
> > Thanks
> >
created KIP-769 to expose connector configuration definitions in
> the Connect API
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
>
> Please take a look and let me know if you have any feedback.
>
> Thanks
>
Hi,
I've created KIP-769 to expose connector configuration definitions in
the Connect API
https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
Please take a look and let me know if you have any feedback.
Thanks
35 matches
Mail list logo
