> 1) it's more readable to separete them into different keys, because they are orthogonal concepts. I'm not sure whether this is more clear, but using one key will be definitely one line less.
> 2) the planner can give all the availble versions in the exception message, if user uses a wrong version (this is often reported in user ML). Lists all available connectors seems also quite straightforward, e.g user provided a wrong "kafka-0.8", we tell user we have candidates of "kakfa-0.11", "kafka-universal" > 3) we may have to write a full documentation for each version, because they are different "connector"? I don't think so. We can still treat it as the same connector but with different versions. > 4) At some point in the future, we may don't need users to specify kafka version anymore. I think this could make user confuse. For example, let's say current "universal" kafka connector is the unified connector, which can support all kinds of versions. Then users will have different experience regarding to version: If you provide "version=0.11", we will use "kafka-0.11" for connector. But if users don't set any version, we will use "kafka-universal" instead. The behavior is inconsistent IMO. Best, Kurt On Mon, Mar 30, 2020 at 3:56 PM Jark Wu <imj...@gmail.com> wrote: > Hi Kurt, > > That's good questions. > > > the meaning of "version" > There are two versions in the old design. One is property version > "connector.property-version" which can be used for backward compatibility. > The other one is "connector.version" which defines the version of external > system, e.g. 0.11" for kafka, "6" or "7" for ES. > In this proposal, the "version" is the previous "connector.version". The > ""connector.property-version" is not introduced in new design. > > > how to keep the old capability which can evolve connector properties > The "connector.property-version" is an optional key in the old design and > is never bumped up. > I'm not sure how "connector.property-version" should work in the initial > design. Maybe @Timo Walther <twal...@apache.org> has more knowledge on > this. > But for the new properties, every options should be expressed as > `ConfigOption` which provides `withDeprecatedKeys(...)` method to easily > support evolving keys. > > > a single keys instead of two, e.g. "kafka-0.11", "kafka-universal"? > There are several benefit to use separate "version" key I can see: > 1) it's more readable to separete them into different keys, because they > are orthogonal concepts. > 2) the planner can give all the availble versions in the exception message, > if user uses a wrong version (this is often reported in user ML). > 3) If we use "kafka-0.11" as connector identifier, we may have to write a > full documentation for each version, because they are different > "connector"? > IMO, for 0.11, 0.11, etc... kafka, they are actually the same connector > but with different "client jar" version, > they share all the same supported property keys and should reside > together. > 4) IMO, the future vision is version-free. At some point in the future, we > may don't need users to specify kafka version anymore, and make > "version=universal" as optional or removed in the future. This is can be > done easily if they are separate keys. > > Best, > Jark > > > On Mon, 30 Mar 2020 at 14:45, Kurt Young <ykt...@gmail.com> wrote: > > > Hi Jark, > > > > Thanks for the proposal, I'm +1 to the general idea. However I have a > > question about "version", > > in the old design, the version seems to be aimed for tracking property > > version, with different > > version, we could evolve these step by step without breaking backward > > compatibility. But in this > > design, version is representing external system's version, like "0.11" > for > > kafka, "6" or "7" for ES. > > I'm not sure if this is necessary, what's the benefit of using two keys > > instead of one, like "kafka-0.11" > > or "ES6" directly? And how about the old capability which could let us > > evolving connector properties? > > > > Best, > > Kurt > > > > > > On Mon, Mar 30, 2020 at 2:36 PM LakeShen <shenleifight...@gmail.com> > > wrote: > > > > > Hi Jark, > > > > > > I am really looking forward to this feature. I think this feature > > > could simplify flink sql code,and at the same time , > > > it could make the developer more easlier to config the flink sql WITH > > > options. > > > > > > Now when I am using flink sql to write flink task , sometimes I think > the > > > WITH options is too long for user. > > > For example,I config the kafka source connector parameter，for consumer > > > group and brokers parameter: > > > > > > 'connector.properties.0.key' = 'group.id' > > > > , 'connector.properties.0.value' = 'xxx' > > > > , 'connector.properties.1.key' = 'bootstrap.servers' > > > > , 'connector.properties.1.value' = 'xxxxx' > > > > > > > > > > I can understand this config , but for the flink fresh man,maybe it > > > is confused for him. > > > In my thought, I am really looking forward to this feature,thank you to > > > propose this feature. > > > > > > Best wishes, > > > LakeShen > > > > > > > > > Jark Wu <imj...@gmail.com> 于2020年3月30日周一 下午2:02写道： > > > > > > > Hi everyone, > > > > > > > > I want to start a discussion about further improve and simplify our > > > current > > > > connector porperty keys, aka WITH options. Currently, we have a > > > > 'connector.' prefix for many properties, but they are verbose, and we > > > see a > > > > big inconsistency between the properties when designing FLIP-107. > > > > > > > > So we propose to remove all the 'connector.' prefix and rename > > > > 'connector.type' to 'connector', 'format.type' to 'format'. So a new > > > Kafka > > > > DDL may look like this: > > > > > > > > CREATE TABLE kafka_table ( > > > > ... > > > > ) WITH ( > > > > 'connector' = 'kafka', > > > > 'version' = '0.10', > > > > 'topic' = 'test-topic', > > > > 'startup-mode' = 'earliest-offset', > > > > 'properties.bootstrap.servers' = 'localhost:9092', > > > > 'properties.group.id' = 'testGroup', > > > > 'format' = 'json', > > > > 'format.fail-on-missing-field' = 'false' > > > > ); > > > > > > > > The new connector property key set will come together with new > Factory > > > > inferface which is proposed in FLIP-95. Old properties are still > > > compatible > > > > with their existing implementation. New properties are only available > > in > > > > new DynamicTableFactory implementations. > > > > > > > > You can access the detailed FLIP here: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory > > > > > > > > Best, > > > > Jark > > > > > > > > > >
