Sorry for resurrecting this thread, but I was trying to build 2.2.0 from source today and noticed there was no v2.2.0 tag (only v2.2.0-RC4). I assume that's not intentional?
On Fri, Dec 8, 2017 at 3:35 AM, Ismaël Mejía <[email protected]> wrote: > Thanks Eugene for opening the poll (sorry if I didn't before I was > quite busy in the last two days but expected to do it today). > > > On Fri, Dec 8, 2017 at 1:27 AM, Ahmet Altay <[email protected]> wrote: > > > > On Thu, Dec 7, 2017 at 3:51 PM, Eugene Kirpichov <[email protected]> > > wrote: > >> > >> I've sent the poll > >> https://lists.apache.org/thread.html/5bc2e184a24de9dbc8184ffd2720d1 > 894010497d47d956b395e037df@%3Cuser.beam.apache.org%3E > >> Will figure out how to tweet from @ApacheBeam, and sent the Twitter poll > >> as well (or ask someone to). > > > > > > I tweeted the poll. > > > >> > >> > >> On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <[email protected]> wrote: > >>> > >>> +1 on moving forward with the plan suggested by kirpichov@ > >>> > >>> On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <[email protected]> > >>> wrote: > >>>> > >>>> +1 to moving forward with this plan. > >>>> > >>>> (FWIW, this seems *less* backwards incompatible than, say, moving from > >>>> Spark 1 to Spark 2, which was decided much quicker. I suppose the > >>>> Spark change has a lower bound on the number of users it could impact > >>>> though.) > >>>> > >>>> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov < > [email protected]> > >>>> wrote: > >>>> > Okay, then let's go forward. Seems that we should: > >>>> > - Open a new poll on user@, in light of 2.2 having been released > >>>> > - Open a twitter poll > >>>> > - Tweet that there's also a poll going on on user@ > >>>> > - Runner authors will reach out to respective runner user > communities > >>>> > - 2 weeks later we gather results and decide > >>>> > ? > >>>> > > >>>> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <[email protected]> > wrote: > >>>> >> > >>>> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far > away, > >>>> >> and starting to improve Beam to offer a Java 8 friendly experience > >>>> >> seems like an excellent idea. > >>>> >> > >>>> >> I understand the backwards compatibility argument. We should do the > >>>> >> poll in twitter + try to reach more users for comments. If you > >>>> >> consider that it is worth, I can open a second poll at user@. > >>>> >> > >>>> >> In any case we should try to move forward, even if we have more > than > >>>> >> 5% of users who want to stay on Java 7 we can consider to maintain > >>>> >> minor releases of a backwards compatible version where we can > >>>> >> backport > >>>> >> only critical fixes e.g. security/data related errors but nothing > >>>> >> new, > >>>> >> in case some user really needs to have them. Of course this can be > >>>> >> some extra work (to be discussed). > >>>> >> > >>>> >> > >>>> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré > >>>> >> <[email protected]> > >>>> >> wrote: > >>>> >> > +1, and sorry again, I thought we got an consensus. > >>>> >> > > >>>> >> > Regards > >>>> >> > JB > >>>> >> > > >>>> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote: > >>>> >> >> > >>>> >> >> +1 to the poll and also to Reuven's point. > >>>> >> >> > >>>> >> >> Those without a support contract would have been using JDK 7 > >>>> >> >> without > >>>> >> >> security updates for years. IMO it seems harmful, as a netizen, > to > >>>> >> >> encourage > >>>> >> >> its use/existence. > >>>> >> >> > >>>> >> >> If there's no noise from the prior thread, then I would assume > no > >>>> >> >> one > >>>> >> >> on > >>>> >> >> user@ has any objection. Anyone else with customers should > reach > >>>> >> >> out to > >>>> >> >> them. > >>>> >> >> > >>>> >> >> Kenn > >>>> >> >> > >>>> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <[email protected] > >>>> >> >> <mailto:[email protected]>> wrote: > >>>> >> >> > >>>> >> >> Technically it's a backwards-incompatible change, however if > >>>> >> >> we are > >>>> >> >> convinced the risk is low we could do it. > >>>> >> >> > >>>> >> >> As mentioned on the original thread, it's not clear that all > >>>> >> >> Beam > >>>> >> >> users read > >>>> >> >> user@ - e.g. most Dataflow users definitely do not. I > think we > >>>> >> >> need > >>>> >> >> to > >>>> >> >> separately reach out to users of each runner through > >>>> >> >> runner-specific > >>>> >> >> channels. > >>>> >> >> > >>>> >> >> Reuven > >>>> >> >> > >>>> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov > >>>> >> >> <[email protected] > >>>> >> >> <mailto:[email protected]>> wrote: > >>>> >> >> > >>>> >> >> On the original thread > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> https://lists.apache.org/thread.html/ > 2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@% > 3Cuser.beam.apache.org%3E > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> <https://lists.apache.org/thread.html/ > 2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@% > 3Cuser.beam.apache.org%3E> > >>>> >> >> , > >>>> >> >> Robert and Ismaël were in favor of no major version > change > >>>> >> >> [Ismaël > >>>> >> >> said:/Also I am afraid that if we wait/ > >>>> >> >> /until we have enough changes to switch Beam to a new > >>>> >> >> major > >>>> >> >> version the > >>>> >> >> switch to Java 8 will happen too late, probably after > Java > >>>> >> >> 8's > >>>> >> >> end > >>>> >> >> of > >>>> >> >> life. And I am not exaggerating, Java 8 is planned to > EOL > >>>> >> >> next > >>>> >> >> march > >>>> >> >> 2018!/]; JB and now Reuven are in favor of a major > version > >>>> >> >> change; > >>>> >> >> > >>>> >> >> nobody so far argued against switching to Java8 in > >>>> >> >> general. > >>>> >> >> > >>>> >> >> I'm personally in favor of no major version change (i.e. > >>>> >> >> not > >>>> >> >> waiting > >>>> >> >> until all other large changes for Beam 3.0 converge, > which > >>>> >> >> will > >>>> >> >> likely > >>>> >> >> be many months), because: > >>>> >> >> - Reasons Ismaël cited; plus the reason that most people > >>>> >> >> are > >>>> >> >> likely > >>>> >> >> already using Java 8. > >>>> >> >> - Going Java8-only earlier will make other Beam 3.0 APIs > >>>> >> >> better > >>>> >> >> for > >>>> >> >> Java8 users, because we (Beam contributors) will have > >>>> >> >> experience > >>>> >> >> working > >>>> >> >> with them within the SDK in Java8 (e.g. writing tests > with > >>>> >> >> use > >>>> >> >> of > >>>> >> >> lambdas and noticing whether it's clunky, or whether > some > >>>> >> >> other > >>>> >> >> Beam > >>>> >> >> APIs need better Java8 support). > >>>> >> >> - Going Java8 will make it more reasonable to include > >>>> >> >> (mostly > >>>> >> >> or > >>>> >> >> only) > >>>> >> >> Java8 snippets in Beam documentation, which will > obviously > >>>> >> >> look > >>>> >> >> more > >>>> >> >> concise and attractive, addressing one of the common > >>>> >> >> concerns > >>>> >> >> of > >>>> >> >> Beam > >>>> >> >> users that it has a heavyweight API compared to > >>>> >> >> functional-style > >>>> >> >> APIs of > >>>> >> >> Spark etc. > >>>> >> >> > >>>> >> >> I think resolving this via a poll of users would be > >>>> >> >> reasonable. > >>>> >> >> I'd > >>>> >> >> suggest e.g. the following phrasing: > >>>> >> >> > >>>> >> >> Apache Beam is considering dropping support for Java 7, > >>>> >> >> and > >>>> >> >> supporting > >>>> >> >> only Java 8 and above in a subsequent release. How would > >>>> >> >> it > >>>> >> >> impact > >>>> >> >> your > >>>> >> >> usage of Beam? > >>>> >> >> - I am already using only Java 8+ for building my Beam > >>>> >> >> code > >>>> >> >> - I am using Java 7 for building my Beam code, but I > would > >>>> >> >> have > >>>> >> >> no > >>>> >> >> trouble switching to Java 8 > >>>> >> >> - I am using Java 7 for building my Beam code, and > >>>> >> >> dropping > >>>> >> >> Java 7 > >>>> >> >> would > >>>> >> >> be a blocker or hindrance to adopting the new release > for > >>>> >> >> me > >>>> >> >> > >>>> >> >> We could tweet this poll on Apache Beam twitter and > >>>> >> >> publish on > >>>> >> >> user@, > >>>> >> >> and, say, if we receive 5% or fewer votes for option 3 > >>>> >> >> after > >>>> >> >> keeping it > >>>> >> >> open for 2 weeks, then adopt Java 8 without a major > >>>> >> >> version > >>>> >> >> change. > >>>> >> >> > >>>> >> >> WDYT? > >>>> >> >> > >>>> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré > >>>> >> >> <[email protected] > >>>> >> >> <mailto:[email protected]>> wrote: > >>>> >> >> > >>>> >> >> Good idea ! Definitely +1 > >>>> >> >> > >>>> >> >> Regards > >>>> >> >> JB > >>>> >> >> > >>>> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote: > >>>> >> >> > We should bring this up on the Beam 3.0 thread. > >>>> >> >> Since > >>>> >> >> it's > >>>> >> >> technically a > >>>> >> >> > backwards-incompatible change, it might make a > good > >>>> >> >> item > >>>> >> >> for Beam > >>>> >> >> 3.0. > >>>> >> >> > > >>>> >> >> > Reuven > >>>> >> >> > > >>>> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste > >>>> >> >> Onofré > >>>> >> >> <[email protected] <mailto:[email protected]> > >>>> >> >> > <mailto:[email protected] <mailto:[email protected] > >>> > >>>> >> >> wrote: > >>>> >> >> > > >>>> >> >> > My apologizes, I thought we had a consensus > >>>> >> >> already. > >>>> >> >> > > >>>> >> >> > Regards > >>>> >> >> > JB > >>>> >> >> > > >>>> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov > wrote: > >>>> >> >> > > >>>> >> >> > Thanks JB for sending the detailed notes > >>>> >> >> about > >>>> >> >> new > >>>> >> >> stuff > >>>> >> >> in 2.2.0! A lot > >>>> >> >> > of exciting things indeed. > >>>> >> >> > > >>>> >> >> > Regarding Java 8: I thought our consensus > >>>> >> >> was to > >>>> >> >> have the > >>>> >> >> release notes > >>>> >> >> > say that we're *considering* going > >>>> >> >> Java8-only, > >>>> >> >> and > >>>> >> >> use > >>>> >> >> that to get more > >>>> >> >> > opinions from the user community - but I > >>>> >> >> can't > >>>> >> >> find > >>>> >> >> the > >>>> >> >> emails that made > >>>> >> >> > me think so. > >>>> >> >> > > >>>> >> >> > +Ismaël Mejía <mailto:[email protected] > >>>> >> >> <mailto:[email protected]> <mailto: > [email protected] > >>>> >> >> <mailto:[email protected]>>> - do > >>>> >> >> > you think we should formally conclude the > >>>> >> >> vote > >>>> >> >> on > >>>> >> >> the > >>>> >> >> thread [VOTE] > >>>> >> >> > [DISCUSSION] Remove support for Java 7? > >>>> >> >> > Or should we take more steps - e.g. > perhaps > >>>> >> >> tweet a > >>>> >> >> link > >>>> >> >> to that thread > >>>> >> >> > from the Beam twitter account, ask people > >>>> >> >> to > >>>> >> >> chime > >>>> >> >> in, > >>>> >> >> and wait for say > >>>> >> >> > 2 weeks before declaring a conclusion? > >>>> >> >> > > >>>> >> >> > Let's also have a process JIRA for going > >>>> >> >> Java8. > >>>> >> >> I've > >>>> >> >> filed one: > >>>> >> >> > https://issues.apache.org/jira/browse/BEAM-3285 > >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285> > >>>> >> >> > > >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285 > >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>> > >>>> >> >> > > >>>> >> >> > On Mon, Dec 4, 2017 at 1:58 AM > >>>> >> >> Jean-Baptiste > >>>> >> >> Onofré > >>>> >> >> <[email protected] <mailto:[email protected]> > >>>> >> >> > <mailto:[email protected] > >>>> >> >> <mailto:[email protected]>> > >>>> >> >> <mailto:[email protected] <mailto:[email protected]> > >>>> >> >> > <mailto:[email protected] > >>>> >> >> <mailto:[email protected]>>>> > >>>> >> >> wrote: > >>>> >> >> > > >>>> >> >> > Just an important note that we > forgot > >>>> >> >> to > >>>> >> >> mention. > >>>> >> >> > > >>>> >> >> > !! The 2.2.0 release will be the > last > >>>> >> >> one > >>>> >> >> supporting > >>>> >> >> Spark 1.x and > >>>> >> >> > Java 7 !! > >>>> >> >> > > >>>> >> >> > Starting from Beam 2.3.0, the Spark > >>>> >> >> runner > >>>> >> >> will work > >>>> >> >> only with > >>>> >> >> > Spark 2.x and we > >>>> >> >> > will focus only Java 8. > >>>> >> >> > > >>>> >> >> > Regards > >>>> >> >> > JB > >>>> >> >> > > >>>> >> >> > On 12/04/2017 10:15 AM, > Jean-Baptiste > >>>> >> >> Onofré > >>>> >> >> wrote: > >>>> >> >> > > Thanks Reuven ! > >>>> >> >> > > > >>>> >> >> > > I would like to emphasize on some > >>>> >> >> highlights in > >>>> >> >> 2.2.0 release: > >>>> >> >> > > > >>>> >> >> > > - New IOs have been introduced: > >>>> >> >> > > * TikaIO leveraging Apache > Tika, > >>>> >> >> allowing > >>>> >> >> the > >>>> >> >> deal with a lot > >>>> >> >> > of different > >>>> >> >> > > data formats > >>>> >> >> > > * RedisIO to read and write > >>>> >> >> key/value > >>>> >> >> pairs > >>>> >> >> from a Redis > >>>> >> >> > server. This > >>>> >> >> > IO will > >>>> >> >> > > be soon extended to Redis PubSub. > >>>> >> >> > > * FileIO provides transforms > for > >>>> >> >> working > >>>> >> >> with > >>>> >> >> files (raw). > >>>> >> >> > Especially, it > >>>> >> >> > > provides matching file patterns > and > >>>> >> >> read > >>>> >> >> on > >>>> >> >> patterns. It can be > >>>> >> >> > easily > >>>> >> >> > extended > >>>> >> >> > > for a specific format (like we do > >>>> >> >> in > >>>> >> >> AvroIO > >>>> >> >> or > >>>> >> >> TextIO now). > >>>> >> >> > > * SolrIO to interact with > Apache > >>>> >> >> Solr > >>>> >> >> (Lucene) > >>>> >> >> > > > >>>> >> >> > > - On the other hand, improvements > >>>> >> >> have > >>>> >> >> been > >>>> >> >> performed on > >>>> >> >> > existing IOs: > >>>> >> >> > > * We started to introduce > readAll > >>>> >> >> pattern > >>>> >> >> in > >>>> >> >> IOs (AvroIO, > >>>> >> >> > TextIO, JdbcIO, > >>>> >> >> > > ...), allowing to pass "request" > >>>> >> >> arguments > >>>> >> >> via an > >>>> >> >> input PCollection. > >>>> >> >> > > * ElasticsearchIO has an > improved > >>>> >> >> support > >>>> >> >> of > >>>> >> >> different > >>>> >> >> > Elasticsearch > >>>> >> >> > version > >>>> >> >> > > (including Elasticsearch 5.x). It > >>>> >> >> also > >>>> >> >> now > >>>> >> >> supports SSL/TLS. > >>>> >> >> > > * HBaseIO is now able to do > >>>> >> >> dynamic > >>>> >> >> work > >>>> >> >> rebalancing > >>>> >> >> > > * KinesisIO uses a more > accurate > >>>> >> >> watermark > >>>> >> >> (based on > >>>> >> >> > approximateArrivalTimestamp) > >>>> >> >> > > * TextIO now supports custom > >>>> >> >> delimiter > >>>> >> >> and like > >>>> >> >> AvroIO, > >>>> >> >> > supports the > >>>> >> >> > readAll > >>>> >> >> > > pattern, > >>>> >> >> > > * Performance improvements on > >>>> >> >> JdbcIO > >>>> >> >> when > >>>> >> >> it > >>>> >> >> has to read lot > >>>> >> >> > of rows > >>>> >> >> > > * Kafka write supports > >>>> >> >> Exactly-Once > >>>> >> >> pattern > >>>> >> >> (introduce in > >>>> >> >> > Kafka 0.11.x) > >>>> >> >> > > > >>>> >> >> > > - A new DSL has been introduced: > >>>> >> >> the SQL > >>>> >> >> DSL ! > >>>> >> >> > > > >>>> >> >> > > We are now focus on 2.3.0 release > >>>> >> >> with > >>>> >> >> new > >>>> >> >> improvements and > >>>> >> >> > features ! > >>>> >> >> > > > >>>> >> >> > > Stay tuned ! > >>>> >> >> > > > >>>> >> >> > > JB on behalf of the Apache Beam > >>>> >> >> community. > >>>> >> >> > > > >>>> >> >> > > On 12/02/2017 11:40 PM, Reuven > Lax > >>>> >> >> wrote: > >>>> >> >> > >> The Apache Beam community is > >>>> >> >> pleased to > >>>> >> >> announce the > >>>> >> >> > availability of the > >>>> >> >> > >> 2.2.0 release. > >>>> >> >> > >> > >>>> >> >> > >> This release adds support for > >>>> >> >> generic > >>>> >> >> file > >>>> >> >> sources and sinks > >>>> >> >> > (beyond TextIO > >>>> >> >> > >> and AvroIO) using FileIO, > >>>> >> >> including > >>>> >> >> support for > >>>> >> >> dynamic > >>>> >> >> > filenames using > >>>> >> >> > >> readAll; this allows streaming > >>>> >> >> pipelines > >>>> >> >> to now > >>>> >> >> read from files by > >>>> >> >> > >> continuously monitoring a > >>>> >> >> directory for > >>>> >> >> new > >>>> >> >> filw. Many other > >>>> >> >> > IOs are > >>>> >> >> > improved, > >>>> >> >> > >> notably including exactly-once > >>>> >> >> support > >>>> >> >> for > >>>> >> >> the > >>>> >> >> Kafka sink. Initial > >>>> >> >> > support for > >>>> >> >> > >> BEAM-SQL is also included in > this > >>>> >> >> release. > >>>> >> >> For a > >>>> >> >> more-complete > >>>> >> >> > list of major > >>>> >> >> > >> changes in the release, please > >>>> >> >> refer to > >>>> >> >> the > >>>> >> >> release notes [2]. > >>>> >> >> > >> > >>>> >> >> > >> The 2.2.0 release is now the > >>>> >> >> recommended > >>>> >> >> version; we encourage > >>>> >> >> > everyone to > >>>> >> >> > >> upgrade from any earlier > releases. > >>>> >> >> > >> > >>>> >> >> > >> We’d like to invite everyone to > >>>> >> >> try out > >>>> >> >> Apache > >>>> >> >> Beam today and > >>>> >> >> > consider > >>>> >> >> > >> joining our vibrant community. > We > >>>> >> >> welcome > >>>> >> >> feedback, > >>>> >> >> > contribution and > >>>> >> >> > >> participation through our > mailing > >>>> >> >> lists, > >>>> >> >> issue > >>>> >> >> tracker, pull > >>>> >> >> > requests, and > >>>> >> >> > >> events. > >>>> >> >> > >> > >>>> >> >> > >> - Reuven Lax, on behalf of the > >>>> >> >> Apache > >>>> >> >> Beam > >>>> >> >> community. > >>>> >> >> > >> > >>>> >> >> > >> [1] > >>>> >> >> https://beam.apache.org/get-started/downloads/ > >>>> >> >> <https://beam.apache.org/get-started/downloads/> > >>>> >> >> > > >>>> >> >> <https://beam.apache.org/get-started/downloads/ > >>>> >> >> <https://beam.apache.org/get-started/downloads/>> > >>>> >> >> > >> [2] > >>>> >> >> > >> > >>>> >> >> > > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12319527&version=12341044 > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12319527&version=12341044> > >>>> >> >> > > >>>> >> >> > >>>> >> >> > >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12319527&version=12341044 > >>>> >> >> > >>>> >> >> > >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12319527&version=12341044>> > >>>> >> >> > >> > >>>> >> >> > > > >>>> >> >> > > >>>> >> >> > -- > >>>> >> >> > Jean-Baptiste Onofré > >>>> >> >> > [email protected] <mailto:[email protected]> > >>>> >> >> <mailto:[email protected] > >>>> >> >> <mailto:[email protected]>> > >>>> >> >> > <mailto:[email protected] > >>>> >> >> <mailto:[email protected]> > >>>> >> >> <mailto:[email protected] > >>>> >> >> <mailto:[email protected]>>> > >>>> >> >> > http://blog.nanthrax.net > >>>> >> >> > Talend - http://www.talend.com > >>>> >> >> > > >>>> >> >> > > >>>> >> >> > -- > >>>> >> >> > Jean-Baptiste Onofré > >>>> >> >> > [email protected] <mailto:[email protected]> > >>>> >> >> <mailto:[email protected] > >>>> >> >> <mailto:[email protected]>> > >>>> >> >> > http://blog.nanthrax.net > >>>> >> >> > Talend - http://www.talend.com > >>>> >> >> > > >>>> >> >> > > >>>> >> >> > >>>> >> >> -- > >>>> >> >> Jean-Baptiste Onofré > >>>> >> >> [email protected] <mailto:[email protected]> > >>>> >> >> http://blog.nanthrax.net > >>>> >> >> Talend - http://www.talend.com > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> > > >>>> >> > -- > >>>> >> > Jean-Baptiste Onofré > >>>> >> > [email protected] > >>>> >> > http://blog.nanthrax.net > >>>> >> > Talend - http://www.talend.com > >>> > >>> > > >
