Hi Dawid, Hi Jark, in my experience it is very important to be able to forward arbitrary properties to the underlying KafkaClient. This seems to be possible in both cases. I am leaning towards Jark's original suggestion. Flink's documentation would only need to state that it forwards everything under "connector.properties" to the KafkaClient.
If the external system has a different formatting scheme, we could still document this on a per-connector basis (use Flink's formatting scheme and document how it is translated to the external system's native properties) or if possible use the external system's scheme under "connector.properties". I don't really know the limitations on our side w.r.t this. Cheers, Konstantin On Wed, Nov 13, 2019 at 3:36 PM Dawid Wysakowicz <dwysakow...@apache.org> wrote: > Hi Jark, > > Majority of the changes make sense to me. I think they will be helpful for > the users. I have a slight concern about the > > kafka's connector.properties property :) . I wonder if we should have > them under a single key: > > connector.properties: > `zookeeper.connect`:`localhost:2181`;`bootstrap.servers`:`localhost:9092` > > As far as I understand it, these are mostly the properties that are > forwarded straight to the underlying KafkaClient. Such properties are > mostly defined and documented by the kafka itself rather than flink. They > might also follow a different formatting scheme than we have for our > properties. Moreover how do we decide which properties are put into the > Properties object and which are not? I would be happy to hear what others > think about that part, as I am not convinced myself about that part. > > Best, > > Dawid > On 13/11/2019 13:22, Jark Wu wrote: > > Hi everyone, > > Connector properties is a very basic component which is used to construct a > connector table via YAML, descriptor API, or DDL. It is also the > serialization representation when persisting into catalog. However, we have > encountered some problems when using connector properties, especially in > DDL. This FLIP is aiming to solve following two issues: > > - FLINK-14645: Data types defined in DDL loses precision and nullability > when converting to properties. > - FLINK-14649: Some properties structure is hard to define in DDL, e.g. > List and Map structure. > > This FLIP proposes new properties to represent data types in schema and to > flatten properties in existing connectors. > > FLIP-86:https://docs.google.com/document/d/14MlXFB45NUmUtesFMFZFjFhd4pqmYamxbfNwUPrfdL4/edit# > > Thanks for any feedback! > > Best, > Jark > > > -- Konstantin Knauf | Solutions Architect +49 160 91394525 Follow us @VervericaData Ververica <https://www.ververica.com/> -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Tony) Cheng
