Hi, Dawid Regarding to `connector.property-version`, I totally agree with you we should implicitly add a "property-version=1" (without 'connector.' prefix) property for future evolving. So I updated FLIP for this. However, I still doubt to use property version to distinguish old/new factory. Because it will break existing DDLs, unless users manually set `connector.property-version=1` to their existing DDLs. So I still prefer to use `connector` vs `connector.type` to distinguish old/new factory.
---------------------------- Hi Timo, +1 to zookeeper.znode-parent > sink.bulk-flush -> sink.buffer-flush? I would like to keep using bulk-flush, because "bulk" is a well-known Elasticsearch API and terminology [1]. I think we don't need to align all the terminologies across Flink connectors. Following the external system's terminology will be more easy-to-understand for connector users. > username -> secrect.username? That's a good idea to hide secret values in logs. However, is there a better way to do that? For example, add a secretOptions() method to Factory? IMO, a `secrect.` prefix is too weak and limit the design of a property key. For example, we want to support authentication for elasticserch [2], a possible property keys will be `authentication.enabled=true`, `authentication.username=jark`, `authentication.password=123456`. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html [2]: https://issues.apache.org/jira/browse/FLINK-16788 ---------------------------- Hi Zhenghua, > does this affect descriptors and related validators? No. As described in the compatiblity section, all the old properties will be routed to the old factories. So all the current descriptors (will be translated to old property keys) are still compatible. But, we should have a plan to translate current descritors into new property keys. However, that is not in the scope of this FLIP and could be done in a separate simple JIRA issue. Best, Jark On Tue, 31 Mar 2020 at 16:08, Zhenghua Gao <doc...@gmail.com> wrote: > Hi Jark, > > Thanks for the proposal. I'm +1 since it's more simple and clear for sql > users. > I have a question about this: does this affect descriptors and related > validators? > > *Best Regards,* > *Zhenghua Gao* > > > On Mon, Mar 30, 2020 at 2:02 PM Jark Wu <imj...@gmail.com> wrote: > > > Hi everyone, > > > > I want to start a discussion about further improve and simplify our > current > > connector porperty keys, aka WITH options. Currently, we have a > > 'connector.' prefix for many properties, but they are verbose, and we > see a > > big inconsistency between the properties when designing FLIP-107. > > > > So we propose to remove all the 'connector.' prefix and rename > > 'connector.type' to 'connector', 'format.type' to 'format'. So a new > Kafka > > DDL may look like this: > > > > CREATE TABLE kafka_table ( > > ... > > ) WITH ( > > 'connector' = 'kafka', > > 'version' = '0.10', > > 'topic' = 'test-topic', > > 'startup-mode' = 'earliest-offset', > > 'properties.bootstrap.servers' = 'localhost:9092', > > 'properties.group.id' = 'testGroup', > > 'format' = 'json', > > 'format.fail-on-missing-field' = 'false' > > ); > > > > The new connector property key set will come together with new Factory > > inferface which is proposed in FLIP-95. Old properties are still > compatible > > with their existing implementation. New properties are only available in > > new DynamicTableFactory implementations. > > > > You can access the detailed FLIP here: > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory > > > > Best, > > Jark > > >
