+1 to moving forward with this plan. (FWIW, this seems *less* backwards incompatible than, say, moving from Spark 1 to Spark 2, which was decided much quicker. I suppose the Spark change has a lower bound on the number of users it could impact though.)
On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <[email protected]> wrote: > Okay, then let's go forward. Seems that we should: > - Open a new poll on user@, in light of 2.2 having been released > - Open a twitter poll > - Tweet that there's also a poll going on on user@ > - Runner authors will reach out to respective runner user communities > - 2 weeks later we gather results and decide > ? > > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <[email protected]> wrote: >> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away, >> and starting to improve Beam to offer a Java 8 friendly experience >> seems like an excellent idea. >> >> I understand the backwards compatibility argument. We should do the >> poll in twitter + try to reach more users for comments. If you >> consider that it is worth, I can open a second poll at user@. >> >> In any case we should try to move forward, even if we have more than >> 5% of users who want to stay on Java 7 we can consider to maintain >> minor releases of a backwards compatible version where we can backport >> only critical fixes e.g. security/data related errors but nothing new, >> in case some user really needs to have them. Of course this can be >> some extra work (to be discussed). >> >> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré <[email protected]> >> wrote: >> > +1, and sorry again, I thought we got an consensus. >> > >> > Regards >> > JB >> > >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote: >> >> >> >> +1 to the poll and also to Reuven's point. >> >> >> >> Those without a support contract would have been using JDK 7 without >> >> security updates for years. IMO it seems harmful, as a netizen, to >> >> encourage >> >> its use/existence. >> >> >> >> If there's no noise from the prior thread, then I would assume no one >> >> on >> >> user@ has any objection. Anyone else with customers should reach out to >> >> them. >> >> >> >> Kenn >> >> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <[email protected] >> >> <mailto:[email protected]>> wrote: >> >> >> >> Technically it's a backwards-incompatible change, however if we are >> >> convinced the risk is low we could do it. >> >> >> >> As mentioned on the original thread, it's not clear that all Beam >> >> users read >> >> user@ - e.g. most Dataflow users definitely do not. I think we need >> >> to >> >> separately reach out to users of each runner through >> >> runner-specific >> >> channels. >> >> >> >> Reuven >> >> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov >> >> <[email protected] >> >> <mailto:[email protected]>> wrote: >> >> >> >> On the original thread >> >> >> >> >> >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E >> >> >> >> >> >> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E> >> >> , >> >> Robert and Ismaël were in favor of no major version change >> >> [Ismaël >> >> said:/Also I am afraid that if we wait/ >> >> /until we have enough changes to switch Beam to a new major >> >> version the >> >> switch to Java 8 will happen too late, probably after Java 8's >> >> end >> >> of >> >> life. And I am not exaggerating, Java 8 is planned to EOL next >> >> march >> >> 2018!/]; JB and now Reuven are in favor of a major version >> >> change; >> >> >> >> nobody so far argued against switching to Java8 in general. >> >> >> >> I'm personally in favor of no major version change (i.e. not >> >> waiting >> >> until all other large changes for Beam 3.0 converge, which will >> >> likely >> >> be many months), because: >> >> - Reasons Ismaël cited; plus the reason that most people are >> >> likely >> >> already using Java 8. >> >> - Going Java8-only earlier will make other Beam 3.0 APIs better >> >> for >> >> Java8 users, because we (Beam contributors) will have >> >> experience >> >> working >> >> with them within the SDK in Java8 (e.g. writing tests with use >> >> of >> >> lambdas and noticing whether it's clunky, or whether some other >> >> Beam >> >> APIs need better Java8 support). >> >> - Going Java8 will make it more reasonable to include (mostly >> >> or >> >> only) >> >> Java8 snippets in Beam documentation, which will obviously look >> >> more >> >> concise and attractive, addressing one of the common concerns >> >> of >> >> Beam >> >> users that it has a heavyweight API compared to >> >> functional-style >> >> APIs of >> >> Spark etc. >> >> >> >> I think resolving this via a poll of users would be reasonable. >> >> I'd >> >> suggest e.g. the following phrasing: >> >> >> >> Apache Beam is considering dropping support for Java 7, and >> >> supporting >> >> only Java 8 and above in a subsequent release. How would it >> >> impact >> >> your >> >> usage of Beam? >> >> - I am already using only Java 8+ for building my Beam code >> >> - I am using Java 7 for building my Beam code, but I would have >> >> no >> >> trouble switching to Java 8 >> >> - I am using Java 7 for building my Beam code, and dropping >> >> Java 7 >> >> would >> >> be a blocker or hindrance to adopting the new release for me >> >> >> >> We could tweet this poll on Apache Beam twitter and publish on >> >> user@, >> >> and, say, if we receive 5% or fewer votes for option 3 after >> >> keeping it >> >> open for 2 weeks, then adopt Java 8 without a major version >> >> change. >> >> >> >> WDYT? >> >> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré >> >> <[email protected] >> >> <mailto:[email protected]>> wrote: >> >> >> >> Good idea ! Definitely +1 >> >> >> >> Regards >> >> JB >> >> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote: >> >> > We should bring this up on the Beam 3.0 thread. Since >> >> it's >> >> technically a >> >> > backwards-incompatible change, it might make a good item >> >> for Beam >> >> 3.0. >> >> > >> >> > Reuven >> >> > >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré >> >> <[email protected] <mailto:[email protected]> >> >> > <mailto:[email protected] <mailto:[email protected]>>> >> >> wrote: >> >> > >> >> > My apologizes, I thought we had a consensus already. >> >> > >> >> > Regards >> >> > JB >> >> > >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote: >> >> > >> >> > Thanks JB for sending the detailed notes about >> >> new >> >> stuff >> >> in 2.2.0! A lot >> >> > of exciting things indeed. >> >> > >> >> > Regarding Java 8: I thought our consensus was to >> >> have the >> >> release notes >> >> > say that we're *considering* going Java8-only, >> >> and >> >> use >> >> that to get more >> >> > opinions from the user community - but I can't >> >> find >> >> the >> >> emails that made >> >> > me think so. >> >> > >> >> > +Ismaël Mejía <mailto:[email protected] >> >> <mailto:[email protected]> <mailto:[email protected] >> >> <mailto:[email protected]>>> - do >> >> > you think we should formally conclude the vote >> >> on >> >> the >> >> thread [VOTE] >> >> > [DISCUSSION] Remove support for Java 7? >> >> > Or should we take more steps - e.g. perhaps >> >> tweet a >> >> link >> >> to that thread >> >> > from the Beam twitter account, ask people to >> >> chime >> >> in, >> >> and wait for say >> >> > 2 weeks before declaring a conclusion? >> >> > >> >> > Let's also have a process JIRA for going Java8. >> >> I've >> >> filed one: >> >> > https://issues.apache.org/jira/browse/BEAM-3285 >> >> <https://issues.apache.org/jira/browse/BEAM-3285> >> >> > <https://issues.apache.org/jira/browse/BEAM-3285 >> >> <https://issues.apache.org/jira/browse/BEAM-3285>> >> >> > >> >> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste >> >> Onofré >> >> <[email protected] <mailto:[email protected]> >> >> > <mailto:[email protected] >> >> <mailto:[email protected]>> >> >> <mailto:[email protected] <mailto:[email protected]> >> >> > <mailto:[email protected] >> >> <mailto:[email protected]>>>> >> >> wrote: >> >> > >> >> > Just an important note that we forgot to >> >> mention. >> >> > >> >> > !! The 2.2.0 release will be the last one >> >> supporting >> >> Spark 1.x and >> >> > Java 7 !! >> >> > >> >> > Starting from Beam 2.3.0, the Spark runner >> >> will work >> >> only with >> >> > Spark 2.x and we >> >> > will focus only Java 8. >> >> > >> >> > Regards >> >> > JB >> >> > >> >> > On 12/04/2017 10:15 AM, Jean-Baptiste >> >> Onofré >> >> wrote: >> >> > > Thanks Reuven ! >> >> > > >> >> > > I would like to emphasize on some >> >> highlights in >> >> 2.2.0 release: >> >> > > >> >> > > - New IOs have been introduced: >> >> > > * TikaIO leveraging Apache Tika, >> >> allowing >> >> the >> >> deal with a lot >> >> > of different >> >> > > data formats >> >> > > * RedisIO to read and write key/value >> >> pairs >> >> from a Redis >> >> > server. This >> >> > IO will >> >> > > be soon extended to Redis PubSub. >> >> > > * FileIO provides transforms for >> >> working >> >> with >> >> files (raw). >> >> > Especially, it >> >> > > provides matching file patterns and read >> >> on >> >> patterns. It can be >> >> > easily >> >> > extended >> >> > > for a specific format (like we do in >> >> AvroIO >> >> or >> >> TextIO now). >> >> > > * SolrIO to interact with Apache Solr >> >> (Lucene) >> >> > > >> >> > > - On the other hand, improvements have >> >> been >> >> performed on >> >> > existing IOs: >> >> > > * We started to introduce readAll >> >> pattern >> >> in >> >> IOs (AvroIO, >> >> > TextIO, JdbcIO, >> >> > > ...), allowing to pass "request" >> >> arguments >> >> via an >> >> input PCollection. >> >> > > * ElasticsearchIO has an improved >> >> support >> >> of >> >> different >> >> > Elasticsearch >> >> > version >> >> > > (including Elasticsearch 5.x). It also >> >> now >> >> supports SSL/TLS. >> >> > > * HBaseIO is now able to do dynamic >> >> work >> >> rebalancing >> >> > > * KinesisIO uses a more accurate >> >> watermark >> >> (based on >> >> > approximateArrivalTimestamp) >> >> > > * TextIO now supports custom delimiter >> >> and like >> >> AvroIO, >> >> > supports the >> >> > readAll >> >> > > pattern, >> >> > > * Performance improvements on JdbcIO >> >> when >> >> it >> >> has to read lot >> >> > of rows >> >> > > * Kafka write supports Exactly-Once >> >> pattern >> >> (introduce in >> >> > Kafka 0.11.x) >> >> > > >> >> > > - A new DSL has been introduced: the SQL >> >> DSL ! >> >> > > >> >> > > We are now focus on 2.3.0 release with >> >> new >> >> improvements and >> >> > features ! >> >> > > >> >> > > Stay tuned ! >> >> > > >> >> > > JB on behalf of the Apache Beam >> >> community. >> >> > > >> >> > > On 12/02/2017 11:40 PM, Reuven Lax >> >> wrote: >> >> > >> The Apache Beam community is pleased to >> >> announce the >> >> > availability of the >> >> > >> 2.2.0 release. >> >> > >> >> >> > >> This release adds support for generic >> >> file >> >> sources and sinks >> >> > (beyond TextIO >> >> > >> and AvroIO) using FileIO, including >> >> support for >> >> dynamic >> >> > filenames using >> >> > >> readAll; this allows streaming >> >> pipelines >> >> to now >> >> read from files by >> >> > >> continuously monitoring a directory for >> >> new >> >> filw. Many other >> >> > IOs are >> >> > improved, >> >> > >> notably including exactly-once support >> >> for >> >> the >> >> Kafka sink. Initial >> >> > support for >> >> > >> BEAM-SQL is also included in this >> >> release. >> >> For a >> >> more-complete >> >> > list of major >> >> > >> changes in the release, please refer to >> >> the >> >> release notes [2]. >> >> > >> >> >> > >> The 2.2.0 release is now the >> >> recommended >> >> version; we encourage >> >> > everyone to >> >> > >> upgrade from any earlier releases. >> >> > >> >> >> > >> We’d like to invite everyone to try out >> >> Apache >> >> Beam today and >> >> > consider >> >> > >> joining our vibrant community. We >> >> welcome >> >> feedback, >> >> > contribution and >> >> > >> participation through our mailing >> >> lists, >> >> issue >> >> tracker, pull >> >> > requests, and >> >> > >> events. >> >> > >> >> >> > >> - Reuven Lax, on behalf of the Apache >> >> Beam >> >> community. >> >> > >> >> >> > >> [1] >> >> https://beam.apache.org/get-started/downloads/ >> >> <https://beam.apache.org/get-started/downloads/> >> >> > <https://beam.apache.org/get-started/downloads/ >> >> <https://beam.apache.org/get-started/downloads/>> >> >> > >> [2] >> >> > >> >> >> > >> >> >> >> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044 >> >> >> >> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044> >> >> > >> >> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044 >> >> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>> >> >> > >> >> >> > > >> >> > >> >> > -- >> >> > Jean-Baptiste Onofré >> >> > [email protected] <mailto:[email protected]> >> >> <mailto:[email protected] <mailto:[email protected]>> >> >> > <mailto:[email protected] >> >> <mailto:[email protected]> >> >> <mailto:[email protected] <mailto:[email protected]>>> >> >> > http://blog.nanthrax.net >> >> > Talend - http://www.talend.com >> >> > >> >> > >> >> > -- >> >> > Jean-Baptiste Onofré >> >> > [email protected] <mailto:[email protected]> >> >> <mailto:[email protected] <mailto:[email protected]>> >> >> > http://blog.nanthrax.net >> >> > Talend - http://www.talend.com >> >> > >> >> > >> >> >> >> -- >> >> Jean-Baptiste Onofré >> >> [email protected] <mailto:[email protected]> >> >> http://blog.nanthrax.net >> >> Talend - http://www.talend.com >> >> >> >> >> >> >> > >> > -- >> > Jean-Baptiste Onofré >> > [email protected] >> > http://blog.nanthrax.net >> > Talend - http://www.talend.com
