+1 on moving forward with the plan suggested by kirpichov@ On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <[email protected]> wrote:
> +1 to moving forward with this plan. > > (FWIW, this seems *less* backwards incompatible than, say, moving from > Spark 1 to Spark 2, which was decided much quicker. I suppose the > Spark change has a lower bound on the number of users it could impact > though.) > > On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <[email protected]> > wrote: > > Okay, then let's go forward. Seems that we should: > > - Open a new poll on user@, in light of 2.2 having been released > > - Open a twitter poll > > - Tweet that there's also a poll going on on user@ > > - Runner authors will reach out to respective runner user communities > > - 2 weeks later we gather results and decide > > ? > > > > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <[email protected]> wrote: > >> > >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away, > >> and starting to improve Beam to offer a Java 8 friendly experience > >> seems like an excellent idea. > >> > >> I understand the backwards compatibility argument. We should do the > >> poll in twitter + try to reach more users for comments. If you > >> consider that it is worth, I can open a second poll at user@. > >> > >> In any case we should try to move forward, even if we have more than > >> 5% of users who want to stay on Java 7 we can consider to maintain > >> minor releases of a backwards compatible version where we can backport > >> only critical fixes e.g. security/data related errors but nothing new, > >> in case some user really needs to have them. Of course this can be > >> some extra work (to be discussed). > >> > >> > >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré <[email protected]> > >> wrote: > >> > +1, and sorry again, I thought we got an consensus. > >> > > >> > Regards > >> > JB > >> > > >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote: > >> >> > >> >> +1 to the poll and also to Reuven's point. > >> >> > >> >> Those without a support contract would have been using JDK 7 without > >> >> security updates for years. IMO it seems harmful, as a netizen, to > >> >> encourage > >> >> its use/existence. > >> >> > >> >> If there's no noise from the prior thread, then I would assume no one > >> >> on > >> >> user@ has any objection. Anyone else with customers should reach > out to > >> >> them. > >> >> > >> >> Kenn > >> >> > >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <[email protected] > >> >> <mailto:[email protected]>> wrote: > >> >> > >> >> Technically it's a backwards-incompatible change, however if we > are > >> >> convinced the risk is low we could do it. > >> >> > >> >> As mentioned on the original thread, it's not clear that all Beam > >> >> users read > >> >> user@ - e.g. most Dataflow users definitely do not. I think we > need > >> >> to > >> >> separately reach out to users of each runner through > >> >> runner-specific > >> >> channels. > >> >> > >> >> Reuven > >> >> > >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov > >> >> <[email protected] > >> >> <mailto:[email protected]>> wrote: > >> >> > >> >> On the original thread > >> >> > >> >> > >> >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130 > fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E > >> >> > >> >> > >> >> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130 > fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E> > >> >> , > >> >> Robert and Ismaël were in favor of no major version change > >> >> [Ismaël > >> >> said:/Also I am afraid that if we wait/ > >> >> /until we have enough changes to switch Beam to a new major > >> >> version the > >> >> switch to Java 8 will happen too late, probably after Java > 8's > >> >> end > >> >> of > >> >> life. And I am not exaggerating, Java 8 is planned to EOL > next > >> >> march > >> >> 2018!/]; JB and now Reuven are in favor of a major version > >> >> change; > >> >> > >> >> nobody so far argued against switching to Java8 in general. > >> >> > >> >> I'm personally in favor of no major version change (i.e. not > >> >> waiting > >> >> until all other large changes for Beam 3.0 converge, which > will > >> >> likely > >> >> be many months), because: > >> >> - Reasons Ismaël cited; plus the reason that most people are > >> >> likely > >> >> already using Java 8. > >> >> - Going Java8-only earlier will make other Beam 3.0 APIs > better > >> >> for > >> >> Java8 users, because we (Beam contributors) will have > >> >> experience > >> >> working > >> >> with them within the SDK in Java8 (e.g. writing tests with > use > >> >> of > >> >> lambdas and noticing whether it's clunky, or whether some > other > >> >> Beam > >> >> APIs need better Java8 support). > >> >> - Going Java8 will make it more reasonable to include (mostly > >> >> or > >> >> only) > >> >> Java8 snippets in Beam documentation, which will obviously > look > >> >> more > >> >> concise and attractive, addressing one of the common concerns > >> >> of > >> >> Beam > >> >> users that it has a heavyweight API compared to > >> >> functional-style > >> >> APIs of > >> >> Spark etc. > >> >> > >> >> I think resolving this via a poll of users would be > reasonable. > >> >> I'd > >> >> suggest e.g. the following phrasing: > >> >> > >> >> Apache Beam is considering dropping support for Java 7, and > >> >> supporting > >> >> only Java 8 and above in a subsequent release. How would it > >> >> impact > >> >> your > >> >> usage of Beam? > >> >> - I am already using only Java 8+ for building my Beam code > >> >> - I am using Java 7 for building my Beam code, but I would > have > >> >> no > >> >> trouble switching to Java 8 > >> >> - I am using Java 7 for building my Beam code, and dropping > >> >> Java 7 > >> >> would > >> >> be a blocker or hindrance to adopting the new release for me > >> >> > >> >> We could tweet this poll on Apache Beam twitter and publish > on > >> >> user@, > >> >> and, say, if we receive 5% or fewer votes for option 3 after > >> >> keeping it > >> >> open for 2 weeks, then adopt Java 8 without a major version > >> >> change. > >> >> > >> >> WDYT? > >> >> > >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré > >> >> <[email protected] > >> >> <mailto:[email protected]>> wrote: > >> >> > >> >> Good idea ! Definitely +1 > >> >> > >> >> Regards > >> >> JB > >> >> > >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote: > >> >> > We should bring this up on the Beam 3.0 thread. Since > >> >> it's > >> >> technically a > >> >> > backwards-incompatible change, it might make a good > item > >> >> for Beam > >> >> 3.0. > >> >> > > >> >> > Reuven > >> >> > > >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré > >> >> <[email protected] <mailto:[email protected]> > >> >> > <mailto:[email protected] <mailto:[email protected]>>> > >> >> wrote: > >> >> > > >> >> > My apologizes, I thought we had a consensus > already. > >> >> > > >> >> > Regards > >> >> > JB > >> >> > > >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote: > >> >> > > >> >> > Thanks JB for sending the detailed notes about > >> >> new > >> >> stuff > >> >> in 2.2.0! A lot > >> >> > of exciting things indeed. > >> >> > > >> >> > Regarding Java 8: I thought our consensus was > to > >> >> have the > >> >> release notes > >> >> > say that we're *considering* going Java8-only, > >> >> and > >> >> use > >> >> that to get more > >> >> > opinions from the user community - but I can't > >> >> find > >> >> the > >> >> emails that made > >> >> > me think so. > >> >> > > >> >> > +Ismaël Mejía <mailto:[email protected] > >> >> <mailto:[email protected]> <mailto:[email protected] > >> >> <mailto:[email protected]>>> - do > >> >> > you think we should formally conclude the vote > >> >> on > >> >> the > >> >> thread [VOTE] > >> >> > [DISCUSSION] Remove support for Java 7? > >> >> > Or should we take more steps - e.g. perhaps > >> >> tweet a > >> >> link > >> >> to that thread > >> >> > from the Beam twitter account, ask people to > >> >> chime > >> >> in, > >> >> and wait for say > >> >> > 2 weeks before declaring a conclusion? > >> >> > > >> >> > Let's also have a process JIRA for going > Java8. > >> >> I've > >> >> filed one: > >> >> > https://issues.apache.org/jira/browse/BEAM-3285 > >> >> <https://issues.apache.org/jira/browse/BEAM-3285> > >> >> > <https://issues.apache.org/ > jira/browse/BEAM-3285 > >> >> <https://issues.apache.org/jira/browse/BEAM-3285>> > >> >> > > >> >> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste > >> >> Onofré > >> >> <[email protected] <mailto:[email protected]> > >> >> > <mailto:[email protected] > >> >> <mailto:[email protected]>> > >> >> <mailto:[email protected] <mailto:[email protected]> > >> >> > <mailto:[email protected] > >> >> <mailto:[email protected]>>>> > >> >> wrote: > >> >> > > >> >> > Just an important note that we forgot to > >> >> mention. > >> >> > > >> >> > !! The 2.2.0 release will be the last one > >> >> supporting > >> >> Spark 1.x and > >> >> > Java 7 !! > >> >> > > >> >> > Starting from Beam 2.3.0, the Spark > runner > >> >> will work > >> >> only with > >> >> > Spark 2.x and we > >> >> > will focus only Java 8. > >> >> > > >> >> > Regards > >> >> > JB > >> >> > > >> >> > On 12/04/2017 10:15 AM, Jean-Baptiste > >> >> Onofré > >> >> wrote: > >> >> > > Thanks Reuven ! > >> >> > > > >> >> > > I would like to emphasize on some > >> >> highlights in > >> >> 2.2.0 release: > >> >> > > > >> >> > > - New IOs have been introduced: > >> >> > > * TikaIO leveraging Apache Tika, > >> >> allowing > >> >> the > >> >> deal with a lot > >> >> > of different > >> >> > > data formats > >> >> > > * RedisIO to read and write > key/value > >> >> pairs > >> >> from a Redis > >> >> > server. This > >> >> > IO will > >> >> > > be soon extended to Redis PubSub. > >> >> > > * FileIO provides transforms for > >> >> working > >> >> with > >> >> files (raw). > >> >> > Especially, it > >> >> > > provides matching file patterns and > read > >> >> on > >> >> patterns. It can be > >> >> > easily > >> >> > extended > >> >> > > for a specific format (like we do in > >> >> AvroIO > >> >> or > >> >> TextIO now). > >> >> > > * SolrIO to interact with Apache > Solr > >> >> (Lucene) > >> >> > > > >> >> > > - On the other hand, improvements have > >> >> been > >> >> performed on > >> >> > existing IOs: > >> >> > > * We started to introduce readAll > >> >> pattern > >> >> in > >> >> IOs (AvroIO, > >> >> > TextIO, JdbcIO, > >> >> > > ...), allowing to pass "request" > >> >> arguments > >> >> via an > >> >> input PCollection. > >> >> > > * ElasticsearchIO has an improved > >> >> support > >> >> of > >> >> different > >> >> > Elasticsearch > >> >> > version > >> >> > > (including Elasticsearch 5.x). It also > >> >> now > >> >> supports SSL/TLS. > >> >> > > * HBaseIO is now able to do dynamic > >> >> work > >> >> rebalancing > >> >> > > * KinesisIO uses a more accurate > >> >> watermark > >> >> (based on > >> >> > approximateArrivalTimestamp) > >> >> > > * TextIO now supports custom > delimiter > >> >> and like > >> >> AvroIO, > >> >> > supports the > >> >> > readAll > >> >> > > pattern, > >> >> > > * Performance improvements on JdbcIO > >> >> when > >> >> it > >> >> has to read lot > >> >> > of rows > >> >> > > * Kafka write supports Exactly-Once > >> >> pattern > >> >> (introduce in > >> >> > Kafka 0.11.x) > >> >> > > > >> >> > > - A new DSL has been introduced: the > SQL > >> >> DSL ! > >> >> > > > >> >> > > We are now focus on 2.3.0 release with > >> >> new > >> >> improvements and > >> >> > features ! > >> >> > > > >> >> > > Stay tuned ! > >> >> > > > >> >> > > JB on behalf of the Apache Beam > >> >> community. > >> >> > > > >> >> > > On 12/02/2017 11:40 PM, Reuven Lax > >> >> wrote: > >> >> > >> The Apache Beam community is pleased > to > >> >> announce the > >> >> > availability of the > >> >> > >> 2.2.0 release. > >> >> > >> > >> >> > >> This release adds support for generic > >> >> file > >> >> sources and sinks > >> >> > (beyond TextIO > >> >> > >> and AvroIO) using FileIO, including > >> >> support for > >> >> dynamic > >> >> > filenames using > >> >> > >> readAll; this allows streaming > >> >> pipelines > >> >> to now > >> >> read from files by > >> >> > >> continuously monitoring a directory > for > >> >> new > >> >> filw. Many other > >> >> > IOs are > >> >> > improved, > >> >> > >> notably including exactly-once > support > >> >> for > >> >> the > >> >> Kafka sink. Initial > >> >> > support for > >> >> > >> BEAM-SQL is also included in this > >> >> release. > >> >> For a > >> >> more-complete > >> >> > list of major > >> >> > >> changes in the release, please refer > to > >> >> the > >> >> release notes [2]. > >> >> > >> > >> >> > >> The 2.2.0 release is now the > >> >> recommended > >> >> version; we encourage > >> >> > everyone to > >> >> > >> upgrade from any earlier releases. > >> >> > >> > >> >> > >> We’d like to invite everyone to try > out > >> >> Apache > >> >> Beam today and > >> >> > consider > >> >> > >> joining our vibrant community. We > >> >> welcome > >> >> feedback, > >> >> > contribution and > >> >> > >> participation through our mailing > >> >> lists, > >> >> issue > >> >> tracker, pull > >> >> > requests, and > >> >> > >> events. > >> >> > >> > >> >> > >> - Reuven Lax, on behalf of the Apache > >> >> Beam > >> >> community. > >> >> > >> > >> >> > >> [1] > >> >> https://beam.apache.org/get-started/downloads/ > >> >> <https://beam.apache.org/get-started/downloads/> > >> >> > <https://beam.apache.org/get- > started/downloads/ > >> >> <https://beam.apache.org/get-started/downloads/>> > >> >> > >> [2] > >> >> > >> > >> >> > > >> >> > >> >> > >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12319527&version=12341044 > >> >> > >> >> > >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12319527&version=12341044> > >> >> > > >> >> > >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12319527&version=12341044 > >> >> > >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12319527&version=12341044>> > >> >> > >> > >> >> > > > >> >> > > >> >> > -- > >> >> > Jean-Baptiste Onofré > >> >> > [email protected] <mailto:[email protected]> > >> >> <mailto:[email protected] <mailto:[email protected] > >> > >> >> > <mailto:[email protected] > >> >> <mailto:[email protected]> > >> >> <mailto:[email protected] <mailto:[email protected] > >>> > >> >> > http://blog.nanthrax.net > >> >> > Talend - http://www.talend.com > >> >> > > >> >> > > >> >> > -- > >> >> > Jean-Baptiste Onofré > >> >> > [email protected] <mailto:[email protected]> > >> >> <mailto:[email protected] <mailto:[email protected] > >> > >> >> > http://blog.nanthrax.net > >> >> > Talend - http://www.talend.com > >> >> > > >> >> > > >> >> > >> >> -- > >> >> Jean-Baptiste Onofré > >> >> [email protected] <mailto:[email protected]> > >> >> http://blog.nanthrax.net > >> >> Talend - http://www.talend.com > >> >> > >> >> > >> >> > >> > > >> > -- > >> > Jean-Baptiste Onofré > >> > [email protected] > >> > http://blog.nanthrax.net > >> > Talend - http://www.talend.com >
