Dear community,
happy to share this week's community digest featuring "Flink Forward
Virtual Conference 2020", a small update on Flink 1.10.1, a better
Filesystem connector for the Table API & SQL, new source/sink interfaces
for the Table API and a bit more.
Flink Development
==============
* [releases] For an update on the outstanding tickets
("Blocker"/"Critical") planned for Apache *Flink 1.10.1* please see the
overview posted by Yu Li in this release discussion thread [1].
* [sql] Timo has shared a proposal (FLIP-95) for *new TableSource and
TableSink interfaces*. It is based on discussions with Jark, Dawid,
Aljoscha, Kurt, Jingsong and many more. Its goals are to simplify the
current interface architecture, to support changelog sources (FLIP-105) and
to remove dependencies on the DataStream API as well as the planner
components. [2]
* [hadoop] Following up on a discussion [3] with Stephan and Till,
Sivaprasanna has shared an overview of Hadoop related utility components to
kick off a discussion on moving these into a separate module
"flink-hadoop-utils". [4]
* [sql] Jingsong Li has started a discussion on introducing a table source
that in essence generates a random stream of data of a given schema to
facilitate development and testing in Flink SQL [5].
* [sql] Jingsong Li has started a discussion on improving the filesystem
connector for the Table API. The current filesystem connector only supports
CSV format and can only be considered experimental for streaming use cases.
There seems to be a consensus to build on top of the existing
StreamingFileSink (DataStream API) and to focus on ORC, Parquet and better
Hive interoperability. [6]
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Flink-1-10-1-tp38689.html
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces
[3]
https://lists.apache.org/thread.html/r198f09496ba46885adbcc41fe778a7a34ad1cd685eeae8beb71e6fbb%40%3Cdev.flink.apache.org%3E
[4]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Introduce-a-new-module-flink-hadoop-utils-tp39107.html
[5]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Introduce-TableFactory-for-StatefulSequenceSource-tp39116.html
[6]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-115-Filesystem-connector-in-Table-tp38870.html
Notable Bugs
==========
* [FLINK-16684] [1.10.0] [1.9.2] The builder of the StreamingFileSink does
not work in Scala. This is one of the blockers to drop support for the
BucketingSink (covered in last week's update). Resolved in Flink 1.10.1. [7]
[7] https://issues.apache.org/jira/browse/FLINK-16684
Events, Blog Posts, Misc
===================
* Unfortunately, we had to cancel Flink Forward SF due to the spread of
SARS-CoV-2 two weeks ago. But instead we will have a three day virtual
Flink Forward conference April 22 - 24. You can register for free under [8]
* Stefan Hausmann has published a blog post on how Apache Flink can be used
for streaming ETL on AWS (Kinesis, Kafka, ElasticSearch and S3
(StreamingFileSink)). [9]
* On the Ververica blog Nico Kruber presents a small benchmark comparing
the overhead of SSL encryption in Flink depending on the SSL provider (JDK
vs OpenSSL). The difference seems to be quite significant. [10]
* Upcoming Meetups: None.
[8] https://www.flink-forward.org/sf-2020
[9]
https://aws.amazon.com/blogs/big-data/streaming-etl-with-apache-flink-and-amazon-kinesis-data-analytics
[10]
https://www.ververica.com/blog/how-openssl-in-ververica-platform-improves-your-flink-job-performance
Cheers,
Konstantin (@snntrable)
--
Konstantin Knauf | Head of Product
+49 160 91394525
Follow us @VervericaData Ververica <https://www.ververica.com/>
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Tony) Cheng
