Just an important note that we forgot to mention.
!! The 2.2.0 release will be the last one supporting Spark 1.x and Java 7 !!
Starting from Beam 2.3.0, the Spark runner will work only with Spark 2.x and we
will focus only Java 8.
Regards
JB
On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote:
Thanks Reuven !
I would like to emphasize on some highlights in 2.2.0 release:
- New IOs have been introduced:
* TikaIO leveraging Apache Tika, allowing the deal with a lot of different
data formats
* RedisIO to read and write key/value pairs from a Redis server. This IO will
be soon extended to Redis PubSub.
* FileIO provides transforms for working with files (raw). Especially, it
provides matching file patterns and read on patterns. It can be easily extended
for a specific format (like we do in AvroIO or TextIO now).
* SolrIO to interact with Apache Solr (Lucene)
- On the other hand, improvements have been performed on existing IOs:
* We started to introduce readAll pattern in IOs (AvroIO, TextIO, JdbcIO,
...), allowing to pass "request" arguments via an input PCollection.
* ElasticsearchIO has an improved support of different Elasticsearch version
(including Elasticsearch 5.x). It also now supports SSL/TLS.
* HBaseIO is now able to do dynamic work rebalancing
* KinesisIO uses a more accurate watermark (based on
approximateArrivalTimestamp)
* TextIO now supports custom delimiter and like AvroIO, supports the readAll
pattern,
* Performance improvements on JdbcIO when it has to read lot of rows
* Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
- A new DSL has been introduced: the SQL DSL !
We are now focus on 2.3.0 release with new improvements and features !
Stay tuned !
JB on behalf of the Apache Beam community.
On 12/02/2017 11:40 PM, Reuven Lax wrote:
The Apache Beam community is pleased to announce the availability of the
2.2.0 release.
This release adds support for generic file sources and sinks (beyond TextIO
and AvroIO) using FileIO, including support for dynamic filenames using
readAll; this allows streaming pipelines to now read from files by
continuously monitoring a directory for new filw. Many other IOs are improved,
notably including exactly-once support for the Kafka sink. Initial support for
BEAM-SQL is also included in this release. For a more-complete list of major
changes in the release, please refer to the release notes [2].
The 2.2.0 release is now the recommended version; we encourage everyone to
upgrade from any earlier releases.
We’d like to invite everyone to try out Apache Beam today and consider
joining our vibrant community. We welcome feedback, contribution and
participation through our mailing lists, issue tracker, pull requests, and
events.
- Reuven Lax, on behalf of the Apache Beam community.
[1] https://beam.apache.org/get-started/downloads/
[2]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
