Dear community, happy to share this week's community digest with an update on Flink 1.8.3, a revival of the n-ary stream operator, a proposal to move our build infrastructure to Azure pipelines, and quite a few other topics. Enjoy.
Flink Development ============== * [releases] The feature freeze for *Flink 1.10* is tonight. * [releases] Hequn has published and started a vote on RC3 for *Flink 1.8.3 *Voting is open until Dec. 10th 2019, 16:00 UTC. No votes so far, but I assume this will change after the feature freeze. [1] * [runtime] Piotr has restarted the discussion to add an *n-ary stream operator* which would help to support multi-broadcast joins in Flink SQL (think of a star schema). The topic has been discussed before (in 2016) in the context of side-inputs and there is already an old design document drafted by Aljoscha to build on top. [2] * [connectors] Chesnay started a conversation to drop support for *Kafka 0.8/0.9* connectors in the upcoming release. It seems that quite a few people are in favor of dropping, but Becket also made a valid point to only deprecate these connector instead of removing them all togehter. [3] * [connectors] Becket has started the vote on *FLIP-27, the new source interface.* This has been a long ongoing topic, but it has not been officially been voted on so far. So there we go. [4,5] * [state backends] Stephan has proposed to drop support for the *synchronous mode of the heap statebackend.* One supporting comment so far. [6] * [hadoop] Craig Foster brought up the topic of *Hadoop 3* support in Apache Flink*.* There is currently no one working on this topic, but Marton Balassi of Cloudera reported that they have been working on this internally and would be willing to contribute their work back next year. [7] * [development process] There is currently no *end-to-end test for Flink deployments on Mesos*. Yangze suggests to such tests now and has started a discussion thread on the topic. He is now looking for feedback from Mesos production users in order to improve the planned test suite. It seems the community will need to maintain their own Mesos docker images for these tests due to Java compatibility issues with the official Mesos docker images. [8] * [development process] Dawid has asked comitters to only vote with their *apache.org <http://apache.org> email* addresses and to only count these votes as binding going forward. Only for apache.org addresses it is possible to verify the status of the voter. [9] * [development process] Following the discussion on reducing the build time of Apache Flink, Robert and others did some experiments to migrate our build from *Travis CI to Azure Pipelines. *In this thread [10] and wiki page [11] he presents his results and asks for opinions on how to move forward. So far there has been a lot of positive feedback to migrate to Azure pipelines mainly motivated by lower build times (due to additional sponsored machines) and richer features. [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-3-release-candidate-3-tp35628.html [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-N-Ary-Stream-Operator-tp11341p35554.html [3] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Drop-Kafka-0-8-0-9-tp35553.html [4] https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface [5] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-27-Refactor-Source-Interface-tp35569.html [6] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Drop-Heap-Backend-Synchronous-snapshots-tp35621.html [7] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Re-Building-with-Hadoop-3-tp35522p35528.html [8] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Adding-e2e-tests-for-Flink-s-Mesos-integration-tp35660p35687.html [9] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Voting-from-apache-org-addresses-tp35499.html [10] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Migrate-build-infrastructure-from-Travis-CI-to-Azure-Pipelines-tp35538.html [11] https://cwiki.apache.org/confluence/display/FLINK/%5Bpreview%5D+Azure+Pipelines Notable Bugs ========== * [FLINK-15063] [1.9.1] The scope of the input/output metrics of the network stack are interchanged, e.g. the outPoolUsage metric can be found under task level scope of "shuffle.netty.input" instead of "shuffle.netty.output". Fixed for 1.9.2. [12] * [FLINK-14949] [1.9.1] [1.8.2] A job can get stuck during cancellation, e.g. if Flink can not spawn the threads, which perform exactly this cancellation. Fixed for 1.9.2 [13] [12] https://issues.apache.org/jira/browse/FLINK-15063 [13] https://issues.apache.org/jira/browse/FLINK-14949 Events, Blog Posts, Misc =================== * *Markos* and *Yuan* have published a recap of Flink Forward Asia 2019 on the Ververica blog including a short summary of the keynotes. [14] * Upcoming Meetups * On December 17th there will be the second Apache Flink meetup in Seoul. [15] *Dongwon* has shared a detailed agenda in last weeks community update. [16] [14] https://www.ververica.com/blog/flink-forward-asia-2019-summary [15] https://www.meetup.com/Seoul-Apache-Flink-Meetup/events/266824815/ [16] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Weekly-Community-Update-2019-48-td35423.html Cheers, Konstantin (@snntrable) -- Konstantin Knauf | Solutions Architect +49 160 91394525 Follow us @VervericaData Ververica <https://www.ververica.com/> -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Tony) Cheng
