Hi all, Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds Barrier Execution Mode for better integration with deep learning frameworks, introduces 30+ built-in and higher-order functions to deal with complex data type easier, improves the K8s integration, along with experimental Scala 2.12 support. Other major updates include the built-in Avro data source, Image data source, flexible streaming sinks, elimination of the 2GB block size limitation during transfer, Pandas UDF improvements. In addition, this release continues to focus on usability, stability, and polish while resolving around 1100 tickets.
We'd like to thank our contributors and users for their contributions and early feedback to this release. This release would not have been possible without you. To download Spark 2.4.0, head over to the download page: http://spark.apache.org/downloads.html To view the release notes: https://spark.apache.org/releases/spark-release-2-4-0.html Thanks, Wenchen PS: If you see any issues with the release notes, webpage or published artifacts, please contact me directly off-list.
