Hey Snehashsih,
Thanks for the KIP. It looks like a very useful feature. Couple of
small-ish points, let me know what you think:
1) Should we update the GET /connectors endpoint to include the version of
the plugin that is running? It could be useful to figure out the version of
the plugin or I am assuming it gets returned by the expand=info call?
2) I am not aware of this and hence asking, can 2 connectors with different
versions have the same name? Does the plugin isolation allow this? This
could have a bearing when using the lifecycle endpoints for connectors like
DELETE etc.
Thanks!
Sagar.
On Tue, Nov 22, 2022 at 2:10 PM Ashwin <apan...@confluent.io.invalid> wrote:
> Hi Snehasis,
>
> > IIUC (please correct me if I am wrong here), what you highlighted above,
> is
> a versioning scheme for a connector config for the same connector (and not
> different versions of a connector plugin).
>
> Sorry for not being more precise in my wording - I meant registering
> versions of schema for connector config.
>
> Let's take the example of a fictional connector which uses a fictional AWS
> service.
>
> Fictional Connector Config schema version:2.0
> ---
> {
> "$schema": "http://json-schema.org/draft-04/schema#";,
> "type": "object",
> "properties": {
> "name": {
> "type": "string"
> },
> "schema_version": {
> "type": "string"
> },
> "aws_access_key": {
> "type": "string"
> },
> "aws_secret_key": {
> "type": "string"
> }
> },
> "required": [
> "name",
> "schema_version",
> "aws_access_key",
> "aws_secret_key"
> ]
> }
>
> Fictional Connector config schema version:3.0
> ---
> {
> "$schema": "http://json-schema.org/draft-04/schema#";,
> "type": "object",
> "properties": {
> "name": {
> "type": "string"
> },
> "schema_version": {
> "type": "string"
> },
> "iam_role": {
> "type": "string"
> }
> },
> "required": [
> "name",
> "schema_version",
> "iam_role"
> ]
> }
>
> The connector which supports Fictional config schema 2.0 will validate the
> access key and secret key.
> Whereas a connector which supports config with schema version 3.0 will only
> validate the IAM role.
>
> This is the alternative which I wanted to suggest. Each plugin will
> register the schema versions of connector config which it supports.
>
> The plugin paths may be optionally different i.e we don't have to
> mandatorily add a new plugin path to support a new schema version.
>
> Thanks,
> Ashwin
>
> On Tue, Nov 22, 2022 at 12:47 PM Snehashis <snehashisp1...@gmail.com>
> wrote:
>
> > Thanks for the input Ashwin.
> >
> > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > config is versioned and has a schema. Then a single plugin (whose
> > > dependencies have not changed) can handle multiple config versions for
> > the
> > > same connector class.
> >
> > IIUC (please correct me if I am wrong here), what you highlighted above,
> is
> > a versioning scheme for a connector config for the same connector (and
> not
> > different versions of a connector plugin). That is a somewhat tangential
> > problem. While it is definitely a useful feature to have, like a log to
> > check what changes were made over time to the config which might make it
> > easier to do rollbacks, it is not the focus here. Here by version we mean
> > to say what underlying version of the plugin should the given
> configuration
> > of the connector use. Perhaps it is better to change the name of the
> > parameter from connector.version to connector.plugin.version or
> > plugin.version if it was confusing. wdyt?
> >
> > > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > > connector/config?migrate=latest", the latest version __attempts__ to
> > > transform the existing config to the newer version. This would require
> > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > connector interface.
> >
> > This is an enhancement we can think of doing in future. Users can simply
> do
> > a PUT call with the updated config which has the updated version number.
> > The assisted mode could be handy as the user does not need to know the
> > config but beyond this it does not seem to justify its existence.
> >
> > Regards
> > Snehashis
> >
> > On Tue, Nov 22, 2022 at 10:50 AM Ashwin <apan...@confluent.io.invalid>
> > wrote:
> >
> > > Hi Snehasis,
> > >
> > > This is a really useful feature and thanks for initiating this
> > discussion.
> > >
> > > I had the following questions -
> > >
> > >
> > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > config is versioned and has a schema. Then a single plugin (whose
> > > dependencies have not changed) can handle multiple config versions for
> > the
> > > same connector class.
> > >
> > > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > > connector/config?migrate=latest", the latest version __attempts__ to
> > > transform the existing config to the newer version. This would require
> > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > connector interface.
> > >
> > > Thanks,
> > > Ashwin
> > >
> > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <snehashisp1...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a discussion thread on KIP-891: Running multiple
> > > versions
> > > > of a connector.
> > > >
> > > > The KIP aims to add the ability for the connect runtime to run
> multiple
> > > > versions of a connector.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > >
> > > > Please take a look and let me know what you think.
> > > >
> > > > Thank you
> > > > Snehashis Pal
> > > >
> > >
> >
>
