Hi Dev We are trying to upgrade the flink version from 1.12.0 to 1.13.2 in apache iceberg project ( https://github.com/apache/iceberg/pull/3116), but it's not a great experience. We expect to support both flink1.12 and flink1.13 in an iceberg-flink module without using the new API of flink1.13 for saving maintenance cost, but we find the iceberg-flink-runtime.jar built by flink 1.13 cannot works fine in flink 1.12 clusters because of the basic API compatibility was break when iterating flink 1.12 to flink1.13.2:
(The following are copied from the iceberg issue: https://github.com/apache/iceberg/issues/3187#issuecomment-928755046) Thanks for the report, @Reo-LEI ! I think this issue was introduced from this apache flink PR ( https://github.com/apache/flink/pull/15316/files#diff-bd276ed951054125b39428ee61de103d9c7832246398f01514a574bb8e51757cR74) and FLINK-21913 (https://issues.apache.org/jira/browse/FLINK-21913), it just changed the returned data type from CatalogTable to ResolvedCatalogTable without any compatibility guarantee. In this case, the iceberg-flink-runtime jar which is compiled from apache flink 1.13 will include the ResovledCatalogTable class inside it. Finally when we package this jar and submit the flink job to flink 1.12, the above compatibility issue happen. As we all know, the DynamicTableFactory ( https://github.com/apache/flink/blob/99c2a415e9eeefafacf70762b6f54070f7911ceb/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java) is a basic API which almost all flink connectors are built on top of it. The breaking compatibility makes the downstream projects really hard to deliver better compatibility to users, unless we iceberg maintain different modules for each maintained flink version (That's not the thing that we want to do). The last flink upgrading work is also not a good experience (See the discussion (https://github.com/apache/iceberg/pull/1956) and comment ( https://github.com/apache/iceberg/pull/1956#discussion_r546534299) ), because the flink 1.12 also breaks several API that was annotated PublicEvolving in flink 1.11.0, that becomes one of the most important reasons leading to the conclusion that stops support flink 1.11.0 in our apache iceberg branch ( Supporting new features [such as flip-27 unified iceberg source/sink] that depends the API introduced in flink 1.12 is another reason). To better support the compatibility of downstream systems and delivering better experience to flink users, I will strongly suggest the Apache Flink community to pay more attention to ensuring API compatibility. Zheng Hu (openinx) Thanks.
