Thanks Jark for the proposal. +1 to the general idea.
For "version", what about "kafka.version"? It is obvious to know its meaning. And I'd like to start a new topic: Should we need to explicitly separate source from sink? With the development of batch and streaming, more and more connectors have both source and sink. So should we set a rule for table properties: - properties for both source and sink: without prefix, like "topic" - properties for source only: with "source." prefix, like "source.startup-mode" - properties for sink only: with "sink." prefix, like "sink.partitioner" What do you think? Best, Jingsong Lee On Mon, Mar 30, 2020 at 3:56 PM Jark Wu <[email protected]> wrote: > Hi Kurt, > > That's good questions. > > > the meaning of "version" > There are two versions in the old design. One is property version > "connector.property-version" which can be used for backward compatibility. > The other one is "connector.version" which defines the version of external > system, e.g. 0.11" for kafka, "6" or "7" for ES. > In this proposal, the "version" is the previous "connector.version". The > ""connector.property-version" is not introduced in new design. > > > how to keep the old capability which can evolve connector properties > The "connector.property-version" is an optional key in the old design and > is never bumped up. > I'm not sure how "connector.property-version" should work in the initial > design. Maybe @Timo Walther <[email protected]> has more knowledge on > this. > But for the new properties, every options should be expressed as > `ConfigOption` which provides `withDeprecatedKeys(...)` method to easily > support evolving keys. > > > a single keys instead of two, e.g. "kafka-0.11", "kafka-universal"? > There are several benefit to use separate "version" key I can see: > 1) it's more readable to separete them into different keys, because they > are orthogonal concepts. > 2) the planner can give all the availble versions in the exception message, > if user uses a wrong version (this is often reported in user ML). > 3) If we use "kafka-0.11" as connector identifier, we may have to write a > full documentation for each version, because they are different > "connector"? > IMO, for 0.11, 0.11, etc... kafka, they are actually the same connector > but with different "client jar" version, > they share all the same supported property keys and should reside > together. > 4) IMO, the future vision is version-free. At some point in the future, we > may don't need users to specify kafka version anymore, and make > "version=universal" as optional or removed in the future. This is can be > done easily if they are separate keys. > > Best, > Jark > > > On Mon, 30 Mar 2020 at 14:45, Kurt Young <[email protected]> wrote: > > > Hi Jark, > > > > Thanks for the proposal, I'm +1 to the general idea. However I have a > > question about "version", > > in the old design, the version seems to be aimed for tracking property > > version, with different > > version, we could evolve these step by step without breaking backward > > compatibility. But in this > > design, version is representing external system's version, like "0.11" > for > > kafka, "6" or "7" for ES. > > I'm not sure if this is necessary, what's the benefit of using two keys > > instead of one, like "kafka-0.11" > > or "ES6" directly? And how about the old capability which could let us > > evolving connector properties? > > > > Best, > > Kurt > > > > > > On Mon, Mar 30, 2020 at 2:36 PM LakeShen <[email protected]> > > wrote: > > > > > Hi Jark, > > > > > > I am really looking forward to this feature. I think this feature > > > could simplify flink sql code,and at the same time , > > > it could make the developer more easlier to config the flink sql WITH > > > options. > > > > > > Now when I am using flink sql to write flink task , sometimes I think > the > > > WITH options is too long for user. > > > For example,I config the kafka source connector parameter，for consumer > > > group and brokers parameter: > > > > > > 'connector.properties.0.key' = 'group.id' > > > > , 'connector.properties.0.value' = 'xxx' > > > > , 'connector.properties.1.key' = 'bootstrap.servers' > > > > , 'connector.properties.1.value' = 'xxxxx' > > > > > > > > > > I can understand this config , but for the flink fresh man,maybe it > > > is confused for him. > > > In my thought, I am really looking forward to this feature,thank you to > > > propose this feature. > > > > > > Best wishes, > > > LakeShen > > > > > > > > > Jark Wu <[email protected]> 于2020年3月30日周一 下午2:02写道： > > > > > > > Hi everyone, > > > > > > > > I want to start a discussion about further improve and simplify our > > > current > > > > connector porperty keys, aka WITH options. Currently, we have a > > > > 'connector.' prefix for many properties, but they are verbose, and we > > > see a > > > > big inconsistency between the properties when designing FLIP-107. > > > > > > > > So we propose to remove all the 'connector.' prefix and rename > > > > 'connector.type' to 'connector', 'format.type' to 'format'. So a new > > > Kafka > > > > DDL may look like this: > > > > > > > > CREATE TABLE kafka_table ( > > > > ... > > > > ) WITH ( > > > > 'connector' = 'kafka', > > > > 'version' = '0.10', > > > > 'topic' = 'test-topic', > > > > 'startup-mode' = 'earliest-offset', > > > > 'properties.bootstrap.servers' = 'localhost:9092', > > > > 'properties.group.id' = 'testGroup', > > > > 'format' = 'json', > > > > 'format.fail-on-missing-field' = 'false' > > > > ); > > > > > > > > The new connector property key set will come together with new > Factory > > > > inferface which is proposed in FLIP-95. Old properties are still > > > compatible > > > > with their existing implementation. New properties are only available > > in > > > > new DynamicTableFactory implementations. > > > > > > > > You can access the detailed FLIP here: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory > > > > > > > > Best, > > > > Jark > > > > > > > > > > -- Best, Jingsong Lee
