Thanks for the explanation Ashwin.
This is an interesting notion. This is something which many connectors
implicitly do anyway. There are several connectors which have different
methods of interpreting the configurations provided. Often the user has
some control over how provided configuration should be used, through
omission of configs, boolean flags that activate/deactivate certain
configs, etc. One could argue that this increases the verbosity of the
configurations and makes it monolithic, however the alternative proposal of
having multiple registered schemas only really seems worthwhile if that the
runtime has the ability to alter the functionality of a connector. There
needs to be some way of registering multiple functionalities, one for each
configuration type. Otherwise, if the runtime is simply passing on the
configuration to the connector, regardless of the which schema version it
belongs to, and delegating the responsibility of picking the functionality
to the connector itself, there is very little the runtime is adding by
registering schemas. Multiple connector versions implicitly define
different configs and functionalities and hence, the ability to run
different versions of the connector itself seems like a more elegant
solution to address this problem.
I also don't think multiple configurations are the only use case for
running different versions of a connector. There could be internal changes
to a connector that do not involve any config changes. A change that
targets a particular enhancement may be incompatible with the older
behaviour. Right now in order to make the changes backwards compatible we
would have to gate the changes behind a connector config (or a different
schema and functionality registration). Otherwise the user is forced to
keep using the older connector until they can upgrade. Problem is if there
are multiple such enhancements (and only one is a breaking change) then
they are missing out on all the other enhements. It is simpler for the user
to have the ability to run both versions of the connector.
On Tue, Nov 22, 2022 at 2:11 PM Ashwin <apan...@confluent.io.invalid> wrote:
> Hi Snehasis,
>
> > IIUC (please correct me if I am wrong here), what you highlighted above,
> is
> a versioning scheme for a connector config for the same connector (and not
> different versions of a connector plugin).
>
> Sorry for not being more precise in my wording - I meant registering
> versions of schema for connector config.
>
> Let's take the example of a fictional connector which uses a fictional AWS
> service.
>
> Fictional Connector Config schema version:2.0
> ---
> {
> "$schema": "http://json-schema.org/draft-04/schema#";,
> "type": "object",
> "properties": {
> "name": {
> "type": "string"
> },
> "schema_version": {
> "type": "string"
> },
> "aws_access_key": {
> "type": "string"
> },
> "aws_secret_key": {
> "type": "string"
> }
> },
> "required": [
> "name",
> "schema_version",
> "aws_access_key",
> "aws_secret_key"
> ]
> }
>
> Fictional Connector config schema version:3.0
> ---
> {
> "$schema": "http://json-schema.org/draft-04/schema#";,
> "type": "object",
> "properties": {
> "name": {
> "type": "string"
> },
> "schema_version": {
> "type": "string"
> },
> "iam_role": {
> "type": "string"
> }
> },
> "required": [
> "name",
> "schema_version",
> "iam_role"
> ]
> }
>
> The connector which supports Fictional config schema 2.0 will validate the
> access key and secret key.
> Whereas a connector which supports config with schema version 3.0 will only
> validate the IAM role.
>
> This is the alternative which I wanted to suggest. Each plugin will
> register the schema versions of connector config which it supports.
>
> The plugin paths may be optionally different i.e we don't have to
> mandatorily add a new plugin path to support a new schema version.
>
> Thanks,
> Ashwin
>
> On Tue, Nov 22, 2022 at 12:47 PM Snehashis <snehashisp1...@gmail.com>
> wrote:
>
> > Thanks for the input Ashwin.
> >
> > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > config is versioned and has a schema. Then a single plugin (whose
> > > dependencies have not changed) can handle multiple config versions for
> > the
> > > same connector class.
> >
> > IIUC (please correct me if I am wrong here), what you highlighted above,
> is
> > a versioning scheme for a connector config for the same connector (and
> not
> > different versions of a connector plugin). That is a somewhat tangential
> > problem. While it is definitely a useful feature to have, like a log to
> > check what changes were made over time to the config which might make it
> > easier to do rollbacks, it is not the focus here. Here by version we mean
> > to say what underlying version of the plugin should the given
> configuration
> > of the connector use. Perhaps it is better to change the name of the
> > parameter from connector.version to connector.plugin.version or
> > plugin.version if it was confusing. wdyt?
> >
> > > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > > connector/config?migrate=latest", the latest version __attempts__ to
> > > transform the existing config to the newer version. This would require
> > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > connector interface.
> >
> > This is an enhancement we can think of doing in future. Users can simply
> do
> > a PUT call with the updated config which has the updated version number.
> > The assisted mode could be handy as the user does not need to know the
> > config but beyond this it does not seem to justify its existence.
> >
> > Regards
> > Snehashis
> >
> > On Tue, Nov 22, 2022 at 10:50 AM Ashwin <apan...@confluent.io.invalid>
> > wrote:
> >
> > > Hi Snehasis,
> > >
> > > This is a really useful feature and thanks for initiating this
> > discussion.
> > >
> > > I had the following questions -
> > >
> > >
> > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > config is versioned and has a schema. Then a single plugin (whose
> > > dependencies have not changed) can handle multiple config versions for
> > the
> > > same connector class.
> > >
> > > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > > connector/config?migrate=latest", the latest version __attempts__ to
> > > transform the existing config to the newer version. This would require
> > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > connector interface.
> > >
> > > Thanks,
> > > Ashwin
> > >
> > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <snehashisp1...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a discussion thread on KIP-891: Running multiple
> > > versions
> > > > of a connector.
> > > >
> > > > The KIP aims to add the ability for the connect runtime to run
> multiple
> > > > versions of a connector.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > >
> > > > Please take a look and let me know what you think.
> > > >
> > > > Thank you
> > > > Snehashis Pal
> > > >
> > >
> >
>
