Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Hi Snehashis, Thank you for the KIP! This is something I've wanted for a long time. I know the discussion has gone cold, are you still interested in pursuing this feature? I'll make time to review the KIP if you are still accepting comments. Thanks, Greg On Tue, Nov 22, 2022 at 12:29 PM Snehashis wrote: > > Thanks for the points Sagar. > > > 1) Should we update the GET /connectors endpoint to include the version of > > the plugin that is running? It could be useful to figure out the version > of > > the plugin or I am assuming it gets returned by the expand=info call? > > I think this is good to have and possible future enhancement. The version > info will be present in the config of the connector if the user has > specified the version. Otherwise it is the latest version which the user > can find out from the connector-plugin endpoint. The information can be > introduced to the response of the GET /connectors endpoint itself, however > the most ideal way of doing this would be to get the currently running > instance of the connector and get the version directly from there. This is > slightly tricky as the connector could be running in a different node. > One way to do this would be to persist the version information in the > status backing store during instantiation of the connector. It requires > some more thought and since the version is part of the configs if provided > and evident otherwise, I have not included it in this KIP. > > > 2) I am not aware of this and hence asking, can 2 connectors with > different > > versions have the same name? Does the plugin isolation allow this? This > > could have a bearing when using the lifecycle endpoints for connectors > like > > DELETE etc. > > All connectors in a cluster need to have uniquire connector names > regardless of what version of the plugin the connector is running > underneath. This is something enforced by the connect runtime itself. All > connect CRUD operations are keyed on the connector name so there will not > be an issue. > > Regards > Snehashis > > On Tue, Nov 22, 2022 at 3:16 PM Sagar wrote: > > > Hey Snehashsih, > > > > Thanks for the KIP. It looks like a very useful feature. Couple of > > small-ish points, let me know what you think: > > > > 1) Should we update the GET /connectors endpoint to include the version of > > the plugin that is running? It could be useful to figure out the version of > > the plugin or I am assuming it gets returned by the expand=info call? > > 2) I am not aware of this and hence asking, can 2 connectors with different > > versions have the same name? Does the plugin isolation allow this? This > > could have a bearing when using the lifecycle endpoints for connectors like > > DELETE etc. > > > > Thanks! > > Sagar. > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin > > wrote: > > > > > Hi Snehasis, > > > > > > > IIUC (please correct me if I am wrong here), what you highlighted > > above, > > > is > > > a versioning scheme for a connector config for the same connector (and > > not > > > different versions of a connector plugin). > > > > > > Sorry for not being more precise in my wording - I meant registering > > > versions of schema for connector config. > > > > > > Let's take the example of a fictional connector which uses a fictional > > AWS > > > service. > > > > > > Fictional Connector Config schema version:2.0 > > > --- > > > { > > > "$schema": "http://json-schema.org/draft-04/schema#;, > > > "type": "object", > > > "properties": { > > > "name": { > > > "type": "string" > > > }, > > > "schema_version": { > > > "type": "string" > > > }, > > > "aws_access_key": { > > > "type": "string" > > > }, > > > "aws_secret_key": { > > > "type": "string" > > > } > > > }, > > > "required": [ > > > "name", > > > "schema_version", > > > "aws_access_key", > > > "aws_secret_key" > > > ] > > > } > > > > > > Fictional Connector config schema version:3.0 > > > --- > > > { > > > "$schema": "http://json-schema.org/draft-04/schema#;, > > > "type": "object", > > > "properties": { > > > "name": { > > > "type": "string" > > > }, > > > "schema_version": { > > > "type": "string" > > > }, > > > "iam_role": { > > > "type": "string" > > > } > > > }, > > > "required": [ > > > "name", > > > "schema_version", > > > "iam_role" > > > ] > > > } > > > > > > The connector which supports Fictional config schema 2.0 will validate > > the > > > access key and secret key. > > > Whereas a connector which supports config with schema version 3.0 will > > only > > > validate the IAM role. > > > > > > This is the alternative which I wanted to suggest. Each plugin will > > > register the schema versions of connector config which it supports. > > > > > > The plugin paths may be optionally different i.e we don't have to > > > mandatorily add a new plugin path to support a new schema version. > > > > > >
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Thanks for the points Sagar. > 1) Should we update the GET /connectors endpoint to include the version of > the plugin that is running? It could be useful to figure out the version of > the plugin or I am assuming it gets returned by the expand=info call? I think this is good to have and possible future enhancement. The version info will be present in the config of the connector if the user has specified the version. Otherwise it is the latest version which the user can find out from the connector-plugin endpoint. The information can be introduced to the response of the GET /connectors endpoint itself, however the most ideal way of doing this would be to get the currently running instance of the connector and get the version directly from there. This is slightly tricky as the connector could be running in a different node. One way to do this would be to persist the version information in the status backing store during instantiation of the connector. It requires some more thought and since the version is part of the configs if provided and evident otherwise, I have not included it in this KIP. > 2) I am not aware of this and hence asking, can 2 connectors with different > versions have the same name? Does the plugin isolation allow this? This > could have a bearing when using the lifecycle endpoints for connectors like > DELETE etc. All connectors in a cluster need to have uniquire connector names regardless of what version of the plugin the connector is running underneath. This is something enforced by the connect runtime itself. All connect CRUD operations are keyed on the connector name so there will not be an issue. Regards Snehashis On Tue, Nov 22, 2022 at 3:16 PM Sagar wrote: > Hey Snehashsih, > > Thanks for the KIP. It looks like a very useful feature. Couple of > small-ish points, let me know what you think: > > 1) Should we update the GET /connectors endpoint to include the version of > the plugin that is running? It could be useful to figure out the version of > the plugin or I am assuming it gets returned by the expand=info call? > 2) I am not aware of this and hence asking, can 2 connectors with different > versions have the same name? Does the plugin isolation allow this? This > could have a bearing when using the lifecycle endpoints for connectors like > DELETE etc. > > Thanks! > Sagar. > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin > wrote: > > > Hi Snehasis, > > > > > IIUC (please correct me if I am wrong here), what you highlighted > above, > > is > > a versioning scheme for a connector config for the same connector (and > not > > different versions of a connector plugin). > > > > Sorry for not being more precise in my wording - I meant registering > > versions of schema for connector config. > > > > Let's take the example of a fictional connector which uses a fictional > AWS > > service. > > > > Fictional Connector Config schema version:2.0 > > --- > > { > > "$schema": "http://json-schema.org/draft-04/schema#;, > > "type": "object", > > "properties": { > > "name": { > > "type": "string" > > }, > > "schema_version": { > > "type": "string" > > }, > > "aws_access_key": { > > "type": "string" > > }, > > "aws_secret_key": { > > "type": "string" > > } > > }, > > "required": [ > > "name", > > "schema_version", > > "aws_access_key", > > "aws_secret_key" > > ] > > } > > > > Fictional Connector config schema version:3.0 > > --- > > { > > "$schema": "http://json-schema.org/draft-04/schema#;, > > "type": "object", > > "properties": { > > "name": { > > "type": "string" > > }, > > "schema_version": { > > "type": "string" > > }, > > "iam_role": { > > "type": "string" > > } > > }, > > "required": [ > > "name", > > "schema_version", > > "iam_role" > > ] > > } > > > > The connector which supports Fictional config schema 2.0 will validate > the > > access key and secret key. > > Whereas a connector which supports config with schema version 3.0 will > only > > validate the IAM role. > > > > This is the alternative which I wanted to suggest. Each plugin will > > register the schema versions of connector config which it supports. > > > > The plugin paths may be optionally different i.e we don't have to > > mandatorily add a new plugin path to support a new schema version. > > > > Thanks, > > Ashwin > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis > > wrote: > > > > > Thanks for the input Ashwin. > > > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector > > > > config is versioned and has a schema. Then a single plugin (whose > > > > dependencies have not changed) can handle multiple config versions > for > > > the > > > > same connector class. > > > > > > IIUC (please correct me if I am wrong here), what you highlighted > above, > > is > > > a versioning scheme for a connector config for the same connector (and > > not > >
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Thanks for the explanation Ashwin. This is an interesting notion. This is something which many connectors implicitly do anyway. There are several connectors which have different methods of interpreting the configurations provided. Often the user has some control over how provided configuration should be used, through omission of configs, boolean flags that activate/deactivate certain configs, etc. One could argue that this increases the verbosity of the configurations and makes it monolithic, however the alternative proposal of having multiple registered schemas only really seems worthwhile if that the runtime has the ability to alter the functionality of a connector. There needs to be some way of registering multiple functionalities, one for each configuration type. Otherwise, if the runtime is simply passing on the configuration to the connector, regardless of the which schema version it belongs to, and delegating the responsibility of picking the functionality to the connector itself, there is very little the runtime is adding by registering schemas. Multiple connector versions implicitly define different configs and functionalities and hence, the ability to run different versions of the connector itself seems like a more elegant solution to address this problem. I also don't think multiple configurations are the only use case for running different versions of a connector. There could be internal changes to a connector that do not involve any config changes. A change that targets a particular enhancement may be incompatible with the older behaviour. Right now in order to make the changes backwards compatible we would have to gate the changes behind a connector config (or a different schema and functionality registration). Otherwise the user is forced to keep using the older connector until they can upgrade. Problem is if there are multiple such enhancements (and only one is a breaking change) then they are missing out on all the other enhements. It is simpler for the user to have the ability to run both versions of the connector. On Tue, Nov 22, 2022 at 2:11 PM Ashwin wrote: > Hi Snehasis, > > > IIUC (please correct me if I am wrong here), what you highlighted above, > is > a versioning scheme for a connector config for the same connector (and not > different versions of a connector plugin). > > Sorry for not being more precise in my wording - I meant registering > versions of schema for connector config. > > Let's take the example of a fictional connector which uses a fictional AWS > service. > > Fictional Connector Config schema version:2.0 > --- > { > "$schema": "http://json-schema.org/draft-04/schema#;, > "type": "object", > "properties": { > "name": { > "type": "string" > }, > "schema_version": { > "type": "string" > }, > "aws_access_key": { > "type": "string" > }, > "aws_secret_key": { > "type": "string" > } > }, > "required": [ > "name", > "schema_version", > "aws_access_key", > "aws_secret_key" > ] > } > > Fictional Connector config schema version:3.0 > --- > { > "$schema": "http://json-schema.org/draft-04/schema#;, > "type": "object", > "properties": { > "name": { > "type": "string" > }, > "schema_version": { > "type": "string" > }, > "iam_role": { > "type": "string" > } > }, > "required": [ > "name", > "schema_version", > "iam_role" > ] > } > > The connector which supports Fictional config schema 2.0 will validate the > access key and secret key. > Whereas a connector which supports config with schema version 3.0 will only > validate the IAM role. > > This is the alternative which I wanted to suggest. Each plugin will > register the schema versions of connector config which it supports. > > The plugin paths may be optionally different i.e we don't have to > mandatorily add a new plugin path to support a new schema version. > > Thanks, > Ashwin > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis > wrote: > > > Thanks for the input Ashwin. > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector > > > config is versioned and has a schema. Then a single plugin (whose > > > dependencies have not changed) can handle multiple config versions for > > the > > > same connector class. > > > > IIUC (please correct me if I am wrong here), what you highlighted above, > is > > a versioning scheme for a connector config for the same connector (and > not > > different versions of a connector plugin). That is a somewhat tangential > > problem. While it is definitely a useful feature to have, like a log to > > check what changes were made over time to the config which might make it > > easier to do rollbacks, it is not the focus here. Here by version we mean > > to say what underlying version of the plugin should the given > configuration > > of the connector use. Perhaps it is better to change the name of the > > parameter from
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Hey Snehashsih, Thanks for the KIP. It looks like a very useful feature. Couple of small-ish points, let me know what you think: 1) Should we update the GET /connectors endpoint to include the version of the plugin that is running? It could be useful to figure out the version of the plugin or I am assuming it gets returned by the expand=info call? 2) I am not aware of this and hence asking, can 2 connectors with different versions have the same name? Does the plugin isolation allow this? This could have a bearing when using the lifecycle endpoints for connectors like DELETE etc. Thanks! Sagar. On Tue, Nov 22, 2022 at 2:10 PM Ashwin wrote: > Hi Snehasis, > > > IIUC (please correct me if I am wrong here), what you highlighted above, > is > a versioning scheme for a connector config for the same connector (and not > different versions of a connector plugin). > > Sorry for not being more precise in my wording - I meant registering > versions of schema for connector config. > > Let's take the example of a fictional connector which uses a fictional AWS > service. > > Fictional Connector Config schema version:2.0 > --- > { > "$schema": "http://json-schema.org/draft-04/schema#;, > "type": "object", > "properties": { > "name": { > "type": "string" > }, > "schema_version": { > "type": "string" > }, > "aws_access_key": { > "type": "string" > }, > "aws_secret_key": { > "type": "string" > } > }, > "required": [ > "name", > "schema_version", > "aws_access_key", > "aws_secret_key" > ] > } > > Fictional Connector config schema version:3.0 > --- > { > "$schema": "http://json-schema.org/draft-04/schema#;, > "type": "object", > "properties": { > "name": { > "type": "string" > }, > "schema_version": { > "type": "string" > }, > "iam_role": { > "type": "string" > } > }, > "required": [ > "name", > "schema_version", > "iam_role" > ] > } > > The connector which supports Fictional config schema 2.0 will validate the > access key and secret key. > Whereas a connector which supports config with schema version 3.0 will only > validate the IAM role. > > This is the alternative which I wanted to suggest. Each plugin will > register the schema versions of connector config which it supports. > > The plugin paths may be optionally different i.e we don't have to > mandatorily add a new plugin path to support a new schema version. > > Thanks, > Ashwin > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis > wrote: > > > Thanks for the input Ashwin. > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector > > > config is versioned and has a schema. Then a single plugin (whose > > > dependencies have not changed) can handle multiple config versions for > > the > > > same connector class. > > > > IIUC (please correct me if I am wrong here), what you highlighted above, > is > > a versioning scheme for a connector config for the same connector (and > not > > different versions of a connector plugin). That is a somewhat tangential > > problem. While it is definitely a useful feature to have, like a log to > > check what changes were made over time to the config which might make it > > easier to do rollbacks, it is not the focus here. Here by version we mean > > to say what underlying version of the plugin should the given > configuration > > of the connector use. Perhaps it is better to change the name of the > > parameter from connector.version to connector.plugin.version or > > plugin.version if it was confusing. wdyt? > > > > > 2. Any plans to support assisted migration e.g if a user invokes "POST > > > connector/config?migrate=latest", the latest version __attempts__ to > > > transform the existing config to the newer version. This would require > > > adding a method like "boolean migrate(Version fromVersion)" to the > > > connector interface. > > > > This is an enhancement we can think of doing in future. Users can simply > do > > a PUT call with the updated config which has the updated version number. > > The assisted mode could be handy as the user does not need to know the > > config but beyond this it does not seem to justify its existence. > > > > Regards > > Snehashis > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin > > wrote: > > > > > Hi Snehasis, > > > > > > This is a really useful feature and thanks for initiating this > > discussion. > > > > > > I had the following questions - > > > > > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector > > > config is versioned and has a schema. Then a single plugin (whose > > > dependencies have not changed) can handle multiple config versions for > > the > > > same connector class. > > > > > > 2. Any plans to support assisted migration e.g if a user invokes "POST > > > connector/config?migrate=latest", the latest version __attempts__ to > > > transform the existing config to the newer version.
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Hi Snehasis, > IIUC (please correct me if I am wrong here), what you highlighted above, is a versioning scheme for a connector config for the same connector (and not different versions of a connector plugin). Sorry for not being more precise in my wording - I meant registering versions of schema for connector config. Let's take the example of a fictional connector which uses a fictional AWS service. Fictional Connector Config schema version:2.0 --- { "$schema": "http://json-schema.org/draft-04/schema#;, "type": "object", "properties": { "name": { "type": "string" }, "schema_version": { "type": "string" }, "aws_access_key": { "type": "string" }, "aws_secret_key": { "type": "string" } }, "required": [ "name", "schema_version", "aws_access_key", "aws_secret_key" ] } Fictional Connector config schema version:3.0 --- { "$schema": "http://json-schema.org/draft-04/schema#;, "type": "object", "properties": { "name": { "type": "string" }, "schema_version": { "type": "string" }, "iam_role": { "type": "string" } }, "required": [ "name", "schema_version", "iam_role" ] } The connector which supports Fictional config schema 2.0 will validate the access key and secret key. Whereas a connector which supports config with schema version 3.0 will only validate the IAM role. This is the alternative which I wanted to suggest. Each plugin will register the schema versions of connector config which it supports. The plugin paths may be optionally different i.e we don't have to mandatorily add a new plugin path to support a new schema version. Thanks, Ashwin On Tue, Nov 22, 2022 at 12:47 PM Snehashis wrote: > Thanks for the input Ashwin. > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector > > config is versioned and has a schema. Then a single plugin (whose > > dependencies have not changed) can handle multiple config versions for > the > > same connector class. > > IIUC (please correct me if I am wrong here), what you highlighted above, is > a versioning scheme for a connector config for the same connector (and not > different versions of a connector plugin). That is a somewhat tangential > problem. While it is definitely a useful feature to have, like a log to > check what changes were made over time to the config which might make it > easier to do rollbacks, it is not the focus here. Here by version we mean > to say what underlying version of the plugin should the given configuration > of the connector use. Perhaps it is better to change the name of the > parameter from connector.version to connector.plugin.version or > plugin.version if it was confusing. wdyt? > > > 2. Any plans to support assisted migration e.g if a user invokes "POST > > connector/config?migrate=latest", the latest version __attempts__ to > > transform the existing config to the newer version. This would require > > adding a method like "boolean migrate(Version fromVersion)" to the > > connector interface. > > This is an enhancement we can think of doing in future. Users can simply do > a PUT call with the updated config which has the updated version number. > The assisted mode could be handy as the user does not need to know the > config but beyond this it does not seem to justify its existence. > > Regards > Snehashis > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin > wrote: > > > Hi Snehasis, > > > > This is a really useful feature and thanks for initiating this > discussion. > > > > I had the following questions - > > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector > > config is versioned and has a schema. Then a single plugin (whose > > dependencies have not changed) can handle multiple config versions for > the > > same connector class. > > > > 2. Any plans to support assisted migration e.g if a user invokes "POST > > connector/config?migrate=latest", the latest version __attempts__ to > > transform the existing config to the newer version. This would require > > adding a method like "boolean migrate(Version fromVersion)" to the > > connector interface. > > > > Thanks, > > Ashwin > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis > > wrote: > > > > > Hi all, > > > > > > I'd like to start a discussion thread on KIP-891: Running multiple > > versions > > > of a connector. > > > > > > The KIP aims to add the ability for the connect runtime to run multiple > > > versions of a connector. > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector > > > > > > Please take a look and let me know what you think. > > > > > > Thank you > > > Snehashis Pal > > > > > >
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Thanks for the input Ashwin. > 1. Can you elaborate on the rejected alternatives ? Suppose connector > config is versioned and has a schema. Then a single plugin (whose > dependencies have not changed) can handle multiple config versions for the > same connector class. IIUC (please correct me if I am wrong here), what you highlighted above, is a versioning scheme for a connector config for the same connector (and not different versions of a connector plugin). That is a somewhat tangential problem. While it is definitely a useful feature to have, like a log to check what changes were made over time to the config which might make it easier to do rollbacks, it is not the focus here. Here by version we mean to say what underlying version of the plugin should the given configuration of the connector use. Perhaps it is better to change the name of the parameter from connector.version to connector.plugin.version or plugin.version if it was confusing. wdyt? > 2. Any plans to support assisted migration e.g if a user invokes "POST > connector/config?migrate=latest", the latest version __attempts__ to > transform the existing config to the newer version. This would require > adding a method like "boolean migrate(Version fromVersion)" to the > connector interface. This is an enhancement we can think of doing in future. Users can simply do a PUT call with the updated config which has the updated version number. The assisted mode could be handy as the user does not need to know the config but beyond this it does not seem to justify its existence. Regards Snehashis On Tue, Nov 22, 2022 at 10:50 AM Ashwin wrote: > Hi Snehasis, > > This is a really useful feature and thanks for initiating this discussion. > > I had the following questions - > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector > config is versioned and has a schema. Then a single plugin (whose > dependencies have not changed) can handle multiple config versions for the > same connector class. > > 2. Any plans to support assisted migration e.g if a user invokes "POST > connector/config?migrate=latest", the latest version __attempts__ to > transform the existing config to the newer version. This would require > adding a method like "boolean migrate(Version fromVersion)" to the > connector interface. > > Thanks, > Ashwin > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis > wrote: > > > Hi all, > > > > I'd like to start a discussion thread on KIP-891: Running multiple > versions > > of a connector. > > > > The KIP aims to add the ability for the connect runtime to run multiple > > versions of a connector. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector > > > > Please take a look and let me know what you think. > > > > Thank you > > Snehashis Pal > > >
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Hi Snehasis, This is a really useful feature and thanks for initiating this discussion. I had the following questions - 1. Can you elaborate on the rejected alternatives ? Suppose connector config is versioned and has a schema. Then a single plugin (whose dependencies have not changed) can handle multiple config versions for the same connector class. 2. Any plans to support assisted migration e.g if a user invokes "POST connector/config?migrate=latest", the latest version __attempts__ to transform the existing config to the newer version. This would require adding a method like "boolean migrate(Version fromVersion)" to the connector interface. Thanks, Ashwin On Mon, Nov 21, 2022 at 2:27 PM Snehashis wrote: > Hi all, > > I'd like to start a discussion thread on KIP-891: Running multiple versions > of a connector. > > The KIP aims to add the ability for the connect runtime to run multiple > versions of a connector. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector > > Please take a look and let me know what you think. > > Thank you > Snehashis Pal >
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Hi Mickael. Thanks for your input. Addressing the point you mentioned below. > 1) Can you explain how this would work with the GET > /{pluginName}/config endpoint? How do you specify a version for a > connector? This API returns the set of configurations for a given connector. Since between versions the configurations can change its allow a user given version to return the correct configs. The version is added as a query parameter, for example - /S3SinkConnector/config?version=v1.1.1. > 2) Some connectors come bundled with transformations (for example > Debezium). How would multiple versions of a transformation be handled? The version of transformations bundled with a particular connector version will be used when the connector is run with the corresponding version number. There will be implicit isolation between the two transformation as they are part of two separate plugins and will be loaded using different plugin classloaders during connector creation. > 3) You mention the latest version will be picked by default if not > specified. The version() method returns a string and currently > enforces no semantics on the value it returns. Can you clarify the new > expected semantics and explain how versions will be compared > (alphabetical, semantics versioning, something else?) The plugin loading mechanism already compares connector versions (new connectors are created with only the latest version though). The comparison between version will remain the same as it is currently and is done using maven artefact versioning plugin. It is a generic versioning scheme that supports semantic, alphabetical and combinations with support additional modifiers (like alpha, beta, release and snapshot build versions). Please refer to this javadoc for the full comparison method. https://maven.apache.org/ref/3.5.2/maven-artifact/apidocs/org/apache/maven/artifact/versioning/ComparableVersion.html . I do not think its necessary to enforce a new semantic with the version. IMO the existing versioning scheme is appropriate and flexible enough for all code versioning methods. On Mon, Nov 21, 2022 at 8:37 PM Mickael Maison wrote: > Hi, > > Thanks for the KIP, this is something that could be really useful! > > 1) Can you explain how this would work with the GET > /{pluginName}/config endpoint? How do you specify a version for a > connector? > > 2) Some connectors come bundled with transformations (for example > Debezium). How would multiple versions of a transformation be handled? > > 3) You mention the latest version will be picked by default if not > specified. The version() method returns a string and currently > enforces no semantics on the value it returns. Can you clarify the new > expected semantics and explain how versions will be compared > (alphabetical, semantics versioning, something else?) > > Thanks, > Mickael > > > On Mon, Nov 21, 2022 at 9:57 AM Snehashis > wrote: > > > > Hi all, > > > > I'd like to start a discussion thread on KIP-891: Running multiple > versions > > of a connector. > > > > The KIP aims to add the ability for the connect runtime to run multiple > > versions of a connector. > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector > > > > Please take a look and let me know what you think. > > > > Thank you > > Snehashis Pal >
RE: Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Hi Mickael. Thanks for your input. Addressing the point you mentioned below. > 1) Can you explain how this would work with the GET > /{pluginName}/config endpoint? How do you specify a version for a > connector? This API returns the set of configurations for a given connector. Since between versions the configurations can change its allow a user given version to return the correct configs. The version is added as a query parameter, for example - /S3SinkConnector/config?version=v1.1.1. > 2) Some connectors come bundled with transformations (for example > Debezium). How would multiple versions of a transformation be handled? The version of transformations bundled with a particular connector version will be used when the connector is run with the corresponding version number. There will be implicit isolation between the two transformation as they are part of two separate plugins and will be loaded using different plugin classloaders during connector creation. > 3) You mention the latest version will be picked by default if not > specified. The version() method returns a string and currently > enforces no semantics on the value it returns. Can you clarify the new > expected semantics and explain how versions will be compared > (alphabetical, semantics versioning, something else?) The plugin loading mechanism already compares connector versions (new connectors are created with only the latest version though). The comparison between version will remain the same as it is currently and is done using maven artefact versioning plugin. It is a generic versioning scheme that supports semantic, alphabetical and combinations with support additional modifiers (like alpha, beta, release and snapshot build versions). Please refer to this javadoc for the full comparison method. https://maven.apache.org/ref/3.5.2/maven-artifact/apidocs/org/apache/maven/artifact/versioning/ComparableVersion.html . I do not think its necessary to enforce a new semantic with the version. IMO the existing versioning scheme is appropriate and flexible enough for all code versioning methods.
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Hi, Thanks for the KIP, this is something that could be really useful! 1) Can you explain how this would work with the GET /{pluginName}/config endpoint? How do you specify a version for a connector? 2) Some connectors come bundled with transformations (for example Debezium). How would multiple versions of a transformation be handled? 3) You mention the latest version will be picked by default if not specified. The version() method returns a string and currently enforces no semantics on the value it returns. Can you clarify the new expected semantics and explain how versions will be compared (alphabetical, semantics versioning, something else?) Thanks, Mickael On Mon, Nov 21, 2022 at 9:57 AM Snehashis wrote: > > Hi all, > > I'd like to start a discussion thread on KIP-891: Running multiple versions > of a connector. > > The KIP aims to add the ability for the connect runtime to run multiple > versions of a connector. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector > > Please take a look and let me know what you think. > > Thank you > Snehashis Pal
[DISCUSS] KIP-891: Running multiple versions of a connector.
Hi all, I'd like to start a discussion thread on KIP-891: Running multiple versions of a connector. The KIP aims to add the ability for the connect runtime to run multiple versions of a connector. https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector Please take a look and let me know what you think. Thank you Snehashis Pal