Hi community,
Not more than one month ago we have released Flink 1.10. We are now heading for the Flink 1.11 release and we, as release managers, would like to share with you what are the features that the community is currently working on and we are hoping that will be part of the Flink 1.11 release. Currently we are aiming with the feature freeze to happen in late April. As for now, some of the features are in the very early stages of the development or even brainstorming. Because of that, some of them do not have associated JIRA tickets or FLIP documents. For the next progress announcement we are hoping that this will be no longer the case. Please also note that because we are still relatively at the beginning of the release cycle, some of the FLIPs haven’t yet been voted. - SQL / Table - FLIP-42: Restructure documentation [1] - FLIP-65: New type inference for Table API UDFs [2] - FLIP-84: Improve TableEnv’s interface [3] - FLIP-91 Introduce SQL client gateway and provide JDBC driver [4] - FLIP-93: Introduce JDBC catalog and Postgres catalog [5] - FLIP-105: Support to interpret and emit changelog in Flink SQL [6] - FLIP-107: Reading table columns from different parts of source records [7] - [FLINK-14807] Add Table#collect API for fetching data [8] - Support query and table hints - ML / Connectors - FLIP-27: New source API [9] - [FLINK-15670] Wrap a source/sink pair to persist intermediate result for subgraph failure recovery [10] - Pulsar source / sink / catalog - Update ML Pipeline API interface to better support Flink ML lib algorithms - PyFlink - FLIP-58: Debugging and monitoring of Python UDF [11] - FLIP-106: Expand the usage scope of Python UDF [12] - Integration with most popular Python libraries (Pandas) - Performance improvements of Python UDF - Support running python UDF in docker workers - Add Python ML API - Fully support all kinds of Python UDF - Web UI - FLIP-98: Better back pressure detection [13] - FLIP-99: Make max exception configurable [14] - FLIP-100: Add attempt information [15] - FLIP-102: Add more metrics to TaskManager [16] - FLIP-103: Better TM/JM log display [17] - [FLINK-14816] Add thread dump feature for TaskManager [18] - Runtime - FLIP-56: Support for dynamic slots on the TaskExecutor [19] - FLIP-67: Support for cluster partitions [20] - FLIP-76: Unaligned checkpoints [21] - FLIP-83: Flink e2e performance testing framework [22] - FLIP-85: Support cluster deploy mode [23] - FLIP-92: Add N-Ary input stream operator in Flink [24] - FLIP-108: Add GPU to the resource management (specifically for UDTF & UDF) [25] - FLIP-111: Consolidate docker images [26] - Unified memory configuration for JobManager - Specify upper bound for number of allocated TaskManagers - [FLINK-9407] ORC format for StreamingFileSink [27] - [FLINK-10742] Let Netty use Flink's buffers on downstream side [28] - [FLINK-10934] Support per-job mode for Kubernetes integration [29] - [FLINK-11395] Avro writer for StreamingFileSink [30] - [FLINK-11427] Protobuf parquet writer for StreamingFileSink [31] - [FLINK-11499] Extend StreamingFileSink BulkFormats to support arbitrary roll policies [32] - [FLINK-14106] Make SlotManager pluggable [33] - [FLINK-15672] Switch to Log4j2 by default [34] - [FLINK-15674] Consolidate Java and Scala type extraction stack [35] - [FLINK-15679] Improve Flink’s ID system [36] - [FLINK-15786] Use the separated classloader to load connectors’ jar [37] - [FLINK-15788] Various Kubernetes improvements [38] - [FLINK-15911][FLINK-15154] Support Flink work over NAT [39] - [FLINK-16408] Bind user code class loader to lifetime of a slot [40] - [FLINK-16428] Network memory management for backpressure [41] - [FLINK-16430] Pipelined region scheduling [42] - Calculate required shuffle memory before allocating slots - State Backend: - [FLINK-5763] Make savepoint self-contained / relocatable [43] - [FLINK-8871] Complete checkpoint cancellation messages [44] - [FLINK-12692] Support disk spilling in HeapKeyedStateBackend [45] - [FLINK-15012] Cleanup of leftover files in HDFS/OSS/S3 [46] - Enable local recovery by default - Support getCustomizedState in KeyedStateStore [47] - [FLINK-15532] Enable strict capacity limit for memory usage for RocksDB [48] Your release managers, Zhijiang & Piotr Nowojski [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs [3] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878 [4] https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway [5] https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog [6] https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL [7] https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Reading+table+columns+from+different+parts+of+source+records [8] https://issues.apache.org/jira/browse/FLINK-14807 [9] https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface [10] https://issues.apache.org/jira/browse/FLINK-15670 [11] https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table [12] https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL [13] https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection [14] https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable [15] https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information [16] https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager [17] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143 [18] https://issues.apache.org/jira/browse/FLINK-14816 [19] https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation [20] https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle [21] https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints [22] https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework [23] https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode [24] https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+Add+N-Ary+Stream+Operator+in+Flink [25] https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink [26] https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification [27] https://issues.apache.org/jira/browse/FLINK-9407 [28] https://issues.apache.org/jira/browse/FLINK-10742 [29] https://issues.apache.org/jira/browse/FLINK-10934 [30] https://issues.apache.org/jira/browse/FLINK-11395 [31] https://issues.apache.org/jira/browse/FLINK-11427 [32] https://issues.apache.org/jira/browse/FLINK-11499 [33] https://issues.apache.org/jira/browse/FLINK-14106 [34] https://issues.apache.org/jira/browse/FLINK-15672 [35] https://issues.apache.org/jira/browse/FLINK-15674 [36] https://issues.apache.org/jira/browse/FLINK-15679 [37] https://issues.apache.org/jira/browse/FLINK-15786 [38] https://issues.apache.org/jira/browse/FLINK-15788 [39] https://issues.apache.org/jira/browse/FLINK-15911 [39] https://issues.apache.org/jira/browse/FLINK-15154 [40] https://issues.apache.org/jira/browse/FLINK-16408 [41] https://issues.apache.org/jira/browse/FLINK-16428 [42] https://issues.apache.org/jira/browse/FLINK-16430 [43] https://issues.apache.org/jira/browse/FLINK-5763 [44] https://issues.apache.org/jira/browse/FLINK-8871 [45] https://issues.apache.org/jira/browse/FLINK-12692 [46] https://issues.apache.org/jira/browse/FLINK-15012 [47] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-customize-state-in-customized-KeyedStateBackend-td32771.html [48] https://issues.apache.org/jira/browse/FLINK-15532
