Hi dev, I'd like to kick off a discussion on adding JDBC catalogs, specifically Postgres catalog in Flink [1].
Currently users have to manually create schemas in Flink source/sink mirroring tables in their relational databases in use cases like JDBC read/write and consuming CDC. Many users have complaint about the unnecessary, redundant, manual work. Any mismatch can lead to a failing Flink job at runtime instead of compile time. All these have been quite unpleasant, resulting in a broken user experience. We want to provide a JDBC catalog interface and a Postgres implementation for Flink as a start to connect to all kinds of relational databases, enabling Flink SQL to 1) retrieve table schema automatically without requiring user writes duped DDL 2) check at compile time for schema errors. It will greatly streamline user experiences when using Flink to deal with popular relational databases like Postgres, MySQL, MariaDB, AWS Aurora, etc. Note that the problem and solution are actually very general to Flink when connecting to all kinds of external systems. We just focus on solving that for relational databases in this FLIP. Thanks, Bowen [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+JDBC+catalog+and+Postgres+catalog
