Bowen Li created FLINK-15985: -------------------------------- Summary: offload runtime params from DDL to table hints in DML/queries Key: FLINK-15985 URL: https://issues.apache.org/jira/browse/FLINK-15985 Project: Flink Issue Type: New Feature Components: Table SQL / API Reporter: Bowen Li Assignee: Danny Chen Fix For: 1.11.0
background: Currently Flink DDL mixes three types of params all together: * External data’s metadata: defines what the data looks like (schema), where it is (location/url), how it should be accessed (username/pwd) * Source/sink runtime params: defines how and usually how fast Flink source/sink reads/writes data, not affecting the results * Kafka “sink-partitioner” * Elastic “bulk-flush.interval/max-size/...” * Semantics params: defines aspects like how much data Flink reads/writes, how the result will look like * Kafka “startup-mode”, “offset” * Watermark, timestamp column Problems of the current mix-up: Flink cannot leverage catalogs and external system metadata alone to run queries with all the non-metadata params involved in DDL. E.g. when we add a catalog for Confluent Schema Registry, the expected user experience should be that Flink users just configure the catalog with url and usr/pwd, and should be able to run queries immediately; however, that’s not the case right now because users still have to use DDL to define a bunch params like “startup-mode”, “offset”, timestamp column, etc, along with the schema redundantly. We’ve heard many user complaints on this. cc [~ykt836] [~lirui] [~lzljs3620320] [~jark] [~twalthr] [~dwysakowicz] -- This message was sent by Atlassian Jira (v8.3.4#803005)