Technically it's a backwards-incompatible change, however if we are convinced the risk is low we could do it.
As mentioned on the original thread, it's not clear that all Beam users read user@ - e.g. most Dataflow users definitely do not. I think we need to separately reach out to users of each runner through runner-specific channels. Reuven On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov <[email protected]> wrote: > On the original thread https://lists.apache.org/thread.html/ > 2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@% > 3Cuser.beam.apache.org%3E , Robert and Ismaël were in favor of no major > version change [Ismaël said:* Also I am afraid that if we wait* > *until we have enough changes to switch Beam to a new major version the > switch to Java 8 will happen too late, probably after Java 8's end of life. > And I am not exaggerating, Java 8 is planned to EOL next march 2018!*]; > JB and now Reuven are in favor of a major version change; nobody so far > argued against switching to Java8 in general. > > I'm personally in favor of no major version change (i.e. not waiting until > all other large changes for Beam 3.0 converge, which will likely be many > months), because: > - Reasons Ismaël cited; plus the reason that most people are likely > already using Java 8. > - Going Java8-only earlier will make other Beam 3.0 APIs better for Java8 > users, because we (Beam contributors) will have experience working with > them within the SDK in Java8 (e.g. writing tests with use of lambdas and > noticing whether it's clunky, or whether some other Beam APIs need better > Java8 support). > - Going Java8 will make it more reasonable to include (mostly or only) > Java8 snippets in Beam documentation, which will obviously look more > concise and attractive, addressing one of the common concerns of Beam users > that it has a heavyweight API compared to functional-style APIs of Spark > etc. > > I think resolving this via a poll of users would be reasonable. I'd > suggest e.g. the following phrasing: > > Apache Beam is considering dropping support for Java 7, and supporting > only Java 8 and above in a subsequent release. How would it impact your > usage of Beam? > - I am already using only Java 8+ for building my Beam code > - I am using Java 7 for building my Beam code, but I would have no trouble > switching to Java 8 > - I am using Java 7 for building my Beam code, and dropping Java 7 would > be a blocker or hindrance to adopting the new release for me > > We could tweet this poll on Apache Beam twitter and publish on user@, > and, say, if we receive 5% or fewer votes for option 3 after keeping it > open for 2 weeks, then adopt Java 8 without a major version change. > > WDYT? > > On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Good idea ! Definitely +1 >> >> Regards >> JB >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote: >> > We should bring this up on the Beam 3.0 thread. Since it's technically a >> > backwards-incompatible change, it might make a good item for Beam 3.0. >> > >> > Reuven >> > >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > My apologizes, I thought we had a consensus already. >> > >> > Regards >> > JB >> > >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote: >> > >> > Thanks JB for sending the detailed notes about new stuff in >> 2.2.0! A lot >> > of exciting things indeed. >> > >> > Regarding Java 8: I thought our consensus was to have the >> release notes >> > say that we're *considering* going Java8-only, and use that to >> get more >> > opinions from the user community - but I can't find the emails >> that made >> > me think so. >> > >> > +Ismaël Mejía <mailto:[email protected] <mailto: >> [email protected]>> - do >> > you think we should formally conclude the vote on the >> thread [VOTE] >> > [DISCUSSION] Remove support for Java 7? >> > Or should we take more steps - e.g. perhaps tweet a link to >> that thread >> > from the Beam twitter account, ask people to chime in, and wait >> for say >> > 2 weeks before declaring a conclusion? >> > >> > Let's also have a process JIRA for going Java8. I've filed one: >> > https://issues.apache.org/jira/browse/BEAM-3285 >> > <https://issues.apache.org/jira/browse/BEAM-3285> >> > >> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste Onofré < >> [email protected] >> > <mailto:[email protected]> <mailto:[email protected] >> > <mailto:[email protected]>>> wrote: >> > >> > Just an important note that we forgot to mention. >> > >> > !! The 2.2.0 release will be the last one supporting Spark >> 1.x and >> > Java 7 !! >> > >> > Starting from Beam 2.3.0, the Spark runner will work only >> with >> > Spark 2.x and we >> > will focus only Java 8. >> > >> > Regards >> > JB >> > >> > On 12/04/2017 10:15 AM, Jean-Baptiste Onofré wrote: >> > > Thanks Reuven ! >> > > >> > > I would like to emphasize on some highlights in 2.2.0 >> release: >> > > >> > > - New IOs have been introduced: >> > > * TikaIO leveraging Apache Tika, allowing the deal >> with a lot >> > of different >> > > data formats >> > > * RedisIO to read and write key/value pairs from a >> Redis >> > server. This >> > IO will >> > > be soon extended to Redis PubSub. >> > > * FileIO provides transforms for working with files >> (raw). >> > Especially, it >> > > provides matching file patterns and read on patterns. >> It can be >> > easily >> > extended >> > > for a specific format (like we do in AvroIO or TextIO >> now). >> > > * SolrIO to interact with Apache Solr (Lucene) >> > > >> > > - On the other hand, improvements have been performed on >> > existing IOs: >> > > * We started to introduce readAll pattern in IOs >> (AvroIO, >> > TextIO, JdbcIO, >> > > ...), allowing to pass "request" arguments via an input >> PCollection. >> > > * ElasticsearchIO has an improved support of different >> > Elasticsearch >> > version >> > > (including Elasticsearch 5.x). It also now supports >> SSL/TLS. >> > > * HBaseIO is now able to do dynamic work rebalancing >> > > * KinesisIO uses a more accurate watermark (based on >> > approximateArrivalTimestamp) >> > > * TextIO now supports custom delimiter and like >> AvroIO, >> > supports the >> > readAll >> > > pattern, >> > > * Performance improvements on JdbcIO when it has to >> read lot >> > of rows >> > > * Kafka write supports Exactly-Once pattern >> (introduce in >> > Kafka 0.11.x) >> > > >> > > - A new DSL has been introduced: the SQL DSL ! >> > > >> > > We are now focus on 2.3.0 release with new improvements >> and >> > features ! >> > > >> > > Stay tuned ! >> > > >> > > JB on behalf of the Apache Beam community. >> > > >> > > On 12/02/2017 11:40 PM, Reuven Lax wrote: >> > >> The Apache Beam community is pleased to announce the >> > availability of the >> > >> 2.2.0 release. >> > >> >> > >> This release adds support for generic file sources and >> sinks >> > (beyond TextIO >> > >> and AvroIO) using FileIO, including support for dynamic >> > filenames using >> > >> readAll; this allows streaming pipelines to now read >> from files by >> > >> continuously monitoring a directory for new filw. Many >> other >> > IOs are >> > improved, >> > >> notably including exactly-once support for the Kafka >> sink. Initial >> > support for >> > >> BEAM-SQL is also included in this release. For a >> more-complete >> > list of major >> > >> changes in the release, please refer to the release >> notes [2]. >> > >> >> > >> The 2.2.0 release is now the recommended version; we >> encourage >> > everyone to >> > >> upgrade from any earlier releases. >> > >> >> > >> We’d like to invite everyone to try out Apache Beam >> today and >> > consider >> > >> joining our vibrant community. We welcome feedback, >> > contribution and >> > >> participation through our mailing lists, issue >> tracker, pull >> > requests, and >> > >> events. >> > >> >> > >> - Reuven Lax, on behalf of the Apache Beam community. >> > >> >> > >> [1] https://beam.apache.org/get-started/downloads/ >> > <https://beam.apache.org/get-started/downloads/> >> > >> [2] >> > >> >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa? >> projectId=12319527&version=12341044 >> > <https://issues.apache.org/jira/secure/ReleaseNote.jspa? >> projectId=12319527&version=12341044> >> > >> >> > > >> > >> > -- >> > Jean-Baptiste Onofré >> > [email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>> >> > http://blog.nanthrax.net >> > Talend - http://www.talend.com >> > >> > >> > -- >> > Jean-Baptiste Onofré >> > [email protected] <mailto:[email protected]> >> > http://blog.nanthrax.net >> > Talend - http://www.talend.com >> > >> > >> >> -- >> Jean-Baptiste Onofré >> [email protected] >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> >
