I've sent the poll https://lists.apache.org/thread.html/5bc2e184a24de9dbc8184ffd2720d1894010497d47d956b395e037df@%3Cuser.beam.apache.org%3E
Will figure out how to tweet from @ApacheBeam, and sent the Twitter poll as well (or ask someone to). On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <[email protected]> wrote: > +1 on moving forward with the plan suggested by kirpichov@ > > On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <[email protected]> > wrote: > >> +1 to moving forward with this plan. >> >> (FWIW, this seems *less* backwards incompatible than, say, moving from >> Spark 1 to Spark 2, which was decided much quicker. I suppose the >> Spark change has a lower bound on the number of users it could impact >> though.) >> >> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <[email protected]> >> wrote: >> > Okay, then let's go forward. Seems that we should: >> > - Open a new poll on user@, in light of 2.2 having been released >> > - Open a twitter poll >> > - Tweet that there's also a poll going on on user@ >> > - Runner authors will reach out to respective runner user communities >> > - 2 weeks later we gather results and decide >> > ? >> > >> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <[email protected]> wrote: >> >> >> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away, >> >> and starting to improve Beam to offer a Java 8 friendly experience >> >> seems like an excellent idea. >> >> >> >> I understand the backwards compatibility argument. We should do the >> >> poll in twitter + try to reach more users for comments. If you >> >> consider that it is worth, I can open a second poll at user@. >> >> >> >> In any case we should try to move forward, even if we have more than >> >> 5% of users who want to stay on Java 7 we can consider to maintain >> >> minor releases of a backwards compatible version where we can backport >> >> only critical fixes e.g. security/data related errors but nothing new, >> >> in case some user really needs to have them. Of course this can be >> >> some extra work (to be discussed). >> >> >> >> >> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré <[email protected]> >> >> wrote: >> >> > +1, and sorry again, I thought we got an consensus. >> >> > >> >> > Regards >> >> > JB >> >> > >> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote: >> >> >> >> >> >> +1 to the poll and also to Reuven's point. >> >> >> >> >> >> Those without a support contract would have been using JDK 7 without >> >> >> security updates for years. IMO it seems harmful, as a netizen, to >> >> >> encourage >> >> >> its use/existence. >> >> >> >> >> >> If there's no noise from the prior thread, then I would assume no >> one >> >> >> on >> >> >> user@ has any objection. Anyone else with customers should reach >> out to >> >> >> them. >> >> >> >> >> >> Kenn >> >> >> >> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <[email protected] >> >> >> <mailto:[email protected]>> wrote: >> >> >> >> >> >> Technically it's a backwards-incompatible change, however if we >> are >> >> >> convinced the risk is low we could do it. >> >> >> >> >> >> As mentioned on the original thread, it's not clear that all >> Beam >> >> >> users read >> >> >> user@ - e.g. most Dataflow users definitely do not. I think we >> need >> >> >> to >> >> >> separately reach out to users of each runner through >> >> >> runner-specific >> >> >> channels. >> >> >> >> >> >> Reuven >> >> >> >> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov >> >> >> <[email protected] >> >> >> <mailto:[email protected]>> wrote: >> >> >> >> >> >> On the original thread >> >> >> >> >> >> >> >> >> >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E >> >> >> >> >> >> >> >> >> < >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E >> > >> >> >> , >> >> >> Robert and Ismaël were in favor of no major version change >> >> >> [Ismaël >> >> >> said:/Also I am afraid that if we wait/ >> >> >> /until we have enough changes to switch Beam to a new major >> >> >> version the >> >> >> switch to Java 8 will happen too late, probably after Java >> 8's >> >> >> end >> >> >> of >> >> >> life. And I am not exaggerating, Java 8 is planned to EOL >> next >> >> >> march >> >> >> 2018!/]; JB and now Reuven are in favor of a major version >> >> >> change; >> >> >> >> >> >> nobody so far argued against switching to Java8 in general. >> >> >> >> >> >> I'm personally in favor of no major version change (i.e. not >> >> >> waiting >> >> >> until all other large changes for Beam 3.0 converge, which >> will >> >> >> likely >> >> >> be many months), because: >> >> >> - Reasons Ismaël cited; plus the reason that most people are >> >> >> likely >> >> >> already using Java 8. >> >> >> - Going Java8-only earlier will make other Beam 3.0 APIs >> better >> >> >> for >> >> >> Java8 users, because we (Beam contributors) will have >> >> >> experience >> >> >> working >> >> >> with them within the SDK in Java8 (e.g. writing tests with >> use >> >> >> of >> >> >> lambdas and noticing whether it's clunky, or whether some >> other >> >> >> Beam >> >> >> APIs need better Java8 support). >> >> >> - Going Java8 will make it more reasonable to include >> (mostly >> >> >> or >> >> >> only) >> >> >> Java8 snippets in Beam documentation, which will obviously >> look >> >> >> more >> >> >> concise and attractive, addressing one of the common >> concerns >> >> >> of >> >> >> Beam >> >> >> users that it has a heavyweight API compared to >> >> >> functional-style >> >> >> APIs of >> >> >> Spark etc. >> >> >> >> >> >> I think resolving this via a poll of users would be >> reasonable. >> >> >> I'd >> >> >> suggest e.g. the following phrasing: >> >> >> >> >> >> Apache Beam is considering dropping support for Java 7, and >> >> >> supporting >> >> >> only Java 8 and above in a subsequent release. How would it >> >> >> impact >> >> >> your >> >> >> usage of Beam? >> >> >> - I am already using only Java 8+ for building my Beam code >> >> >> - I am using Java 7 for building my Beam code, but I would >> have >> >> >> no >> >> >> trouble switching to Java 8 >> >> >> - I am using Java 7 for building my Beam code, and dropping >> >> >> Java 7 >> >> >> would >> >> >> be a blocker or hindrance to adopting the new release for me >> >> >> >> >> >> We could tweet this poll on Apache Beam twitter and publish >> on >> >> >> user@, >> >> >> and, say, if we receive 5% or fewer votes for option 3 after >> >> >> keeping it >> >> >> open for 2 weeks, then adopt Java 8 without a major version >> >> >> change. >> >> >> >> >> >> WDYT? >> >> >> >> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré >> >> >> <[email protected] >> >> >> <mailto:[email protected]>> wrote: >> >> >> >> >> >> Good idea ! Definitely +1 >> >> >> >> >> >> Regards >> >> >> JB >> >> >> >> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote: >> >> >> > We should bring this up on the Beam 3.0 thread. Since >> >> >> it's >> >> >> technically a >> >> >> > backwards-incompatible change, it might make a good >> item >> >> >> for Beam >> >> >> 3.0. >> >> >> > >> >> >> > Reuven >> >> >> > >> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré >> >> >> <[email protected] <mailto:[email protected]> >> >> >> > <mailto:[email protected] <mailto:[email protected]>>> >> >> >> wrote: >> >> >> > >> >> >> > My apologizes, I thought we had a consensus >> already. >> >> >> > >> >> >> > Regards >> >> >> > JB >> >> >> > >> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote: >> >> >> > >> >> >> > Thanks JB for sending the detailed notes >> about >> >> >> new >> >> >> stuff >> >> >> in 2.2.0! A lot >> >> >> > of exciting things indeed. >> >> >> > >> >> >> > Regarding Java 8: I thought our consensus >> was to >> >> >> have the >> >> >> release notes >> >> >> > say that we're *considering* going >> Java8-only, >> >> >> and >> >> >> use >> >> >> that to get more >> >> >> > opinions from the user community - but I >> can't >> >> >> find >> >> >> the >> >> >> emails that made >> >> >> > me think so. >> >> >> > >> >> >> > +Ismaël Mejía <mailto:[email protected] >> >> >> <mailto:[email protected]> <mailto:[email protected] >> >> >> <mailto:[email protected]>>> - do >> >> >> > you think we should formally conclude the >> vote >> >> >> on >> >> >> the >> >> >> thread [VOTE] >> >> >> > [DISCUSSION] Remove support for Java 7? >> >> >> > Or should we take more steps - e.g. perhaps >> >> >> tweet a >> >> >> link >> >> >> to that thread >> >> >> > from the Beam twitter account, ask people to >> >> >> chime >> >> >> in, >> >> >> and wait for say >> >> >> > 2 weeks before declaring a conclusion? >> >> >> > >> >> >> > Let's also have a process JIRA for going >> Java8. >> >> >> I've >> >> >> filed one: >> >> >> > https://issues.apache.org/jira/browse/BEAM-3285 >> >> >> <https://issues.apache.org/jira/browse/BEAM-3285> >> >> >> > < >> https://issues.apache.org/jira/browse/BEAM-3285 >> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>> >> >> >> > >> >> >> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste >> >> >> Onofré >> >> >> <[email protected] <mailto:[email protected]> >> >> >> > <mailto:[email protected] >> >> >> <mailto:[email protected]>> >> >> >> <mailto:[email protected] <mailto:[email protected]> >> >> >> > <mailto:[email protected] >> >> >> <mailto:[email protected]>>>> >> >> >> wrote: >> >> >> > >> >> >> > Just an important note that we forgot to >> >> >> mention. >> >> >> > >> >> >> > !! The 2.2.0 release will be the last >> one >> >> >> supporting >> >> >> Spark 1.x and >> >> >> > Java 7 !! >> >> >> > >> >> >> > Starting from Beam 2.3.0, the Spark >> runner >> >> >> will work >> >> >> only with >> >> >> > Spark 2.x and we >> >> >> > will focus only Java 8. >> >> >> > >> >> >> > Regards >> >> >> > JB >> >> >> > >> >> >> > On 12/04/2017 10:15 AM, Jean-Baptiste >> >> >> Onofré >> >> >> wrote: >> >> >> > > Thanks Reuven ! >> >> >> > > >> >> >> > > I would like to emphasize on some >> >> >> highlights in >> >> >> 2.2.0 release: >> >> >> > > >> >> >> > > - New IOs have been introduced: >> >> >> > > * TikaIO leveraging Apache Tika, >> >> >> allowing >> >> >> the >> >> >> deal with a lot >> >> >> > of different >> >> >> > > data formats >> >> >> > > * RedisIO to read and write >> key/value >> >> >> pairs >> >> >> from a Redis >> >> >> > server. This >> >> >> > IO will >> >> >> > > be soon extended to Redis PubSub. >> >> >> > > * FileIO provides transforms for >> >> >> working >> >> >> with >> >> >> files (raw). >> >> >> > Especially, it >> >> >> > > provides matching file patterns and >> read >> >> >> on >> >> >> patterns. It can be >> >> >> > easily >> >> >> > extended >> >> >> > > for a specific format (like we do in >> >> >> AvroIO >> >> >> or >> >> >> TextIO now). >> >> >> > > * SolrIO to interact with Apache >> Solr >> >> >> (Lucene) >> >> >> > > >> >> >> > > - On the other hand, improvements >> have >> >> >> been >> >> >> performed on >> >> >> > existing IOs: >> >> >> > > * We started to introduce readAll >> >> >> pattern >> >> >> in >> >> >> IOs (AvroIO, >> >> >> > TextIO, JdbcIO, >> >> >> > > ...), allowing to pass "request" >> >> >> arguments >> >> >> via an >> >> >> input PCollection. >> >> >> > > * ElasticsearchIO has an improved >> >> >> support >> >> >> of >> >> >> different >> >> >> > Elasticsearch >> >> >> > version >> >> >> > > (including Elasticsearch 5.x). It >> also >> >> >> now >> >> >> supports SSL/TLS. >> >> >> > > * HBaseIO is now able to do dynamic >> >> >> work >> >> >> rebalancing >> >> >> > > * KinesisIO uses a more accurate >> >> >> watermark >> >> >> (based on >> >> >> > approximateArrivalTimestamp) >> >> >> > > * TextIO now supports custom >> delimiter >> >> >> and like >> >> >> AvroIO, >> >> >> > supports the >> >> >> > readAll >> >> >> > > pattern, >> >> >> > > * Performance improvements on >> JdbcIO >> >> >> when >> >> >> it >> >> >> has to read lot >> >> >> > of rows >> >> >> > > * Kafka write supports Exactly-Once >> >> >> pattern >> >> >> (introduce in >> >> >> > Kafka 0.11.x) >> >> >> > > >> >> >> > > - A new DSL has been introduced: the >> SQL >> >> >> DSL ! >> >> >> > > >> >> >> > > We are now focus on 2.3.0 release >> with >> >> >> new >> >> >> improvements and >> >> >> > features ! >> >> >> > > >> >> >> > > Stay tuned ! >> >> >> > > >> >> >> > > JB on behalf of the Apache Beam >> >> >> community. >> >> >> > > >> >> >> > > On 12/02/2017 11:40 PM, Reuven Lax >> >> >> wrote: >> >> >> > >> The Apache Beam community is >> pleased to >> >> >> announce the >> >> >> > availability of the >> >> >> > >> 2.2.0 release. >> >> >> > >> >> >> >> > >> This release adds support for >> generic >> >> >> file >> >> >> sources and sinks >> >> >> > (beyond TextIO >> >> >> > >> and AvroIO) using FileIO, including >> >> >> support for >> >> >> dynamic >> >> >> > filenames using >> >> >> > >> readAll; this allows streaming >> >> >> pipelines >> >> >> to now >> >> >> read from files by >> >> >> > >> continuously monitoring a directory >> for >> >> >> new >> >> >> filw. Many other >> >> >> > IOs are >> >> >> > improved, >> >> >> > >> notably including exactly-once >> support >> >> >> for >> >> >> the >> >> >> Kafka sink. Initial >> >> >> > support for >> >> >> > >> BEAM-SQL is also included in this >> >> >> release. >> >> >> For a >> >> >> more-complete >> >> >> > list of major >> >> >> > >> changes in the release, please >> refer to >> >> >> the >> >> >> release notes [2]. >> >> >> > >> >> >> >> > >> The 2.2.0 release is now the >> >> >> recommended >> >> >> version; we encourage >> >> >> > everyone to >> >> >> > >> upgrade from any earlier releases. >> >> >> > >> >> >> >> > >> We’d like to invite everyone to try >> out >> >> >> Apache >> >> >> Beam today and >> >> >> > consider >> >> >> > >> joining our vibrant community. We >> >> >> welcome >> >> >> feedback, >> >> >> > contribution and >> >> >> > >> participation through our mailing >> >> >> lists, >> >> >> issue >> >> >> tracker, pull >> >> >> > requests, and >> >> >> > >> events. >> >> >> > >> >> >> >> > >> - Reuven Lax, on behalf of the >> Apache >> >> >> Beam >> >> >> community. >> >> >> > >> >> >> >> > >> [1] >> >> >> https://beam.apache.org/get-started/downloads/ >> >> >> <https://beam.apache.org/get-started/downloads/> >> >> >> > < >> https://beam.apache.org/get-started/downloads/ >> >> >> <https://beam.apache.org/get-started/downloads/>> >> >> >> > >> [2] >> >> >> > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044 >> >> >> >> >> >> >> >> >> < >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044 >> > >> >> >> > >> >> >> >> >> >> < >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044 >> >> >> >> >> >> < >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044 >> >> >> >> >> > >> >> >> >> > > >> >> >> > >> >> >> > -- >> >> >> > Jean-Baptiste Onofré >> >> >> > [email protected] <mailto:[email protected]> >> >> >> <mailto:[email protected] <mailto:[email protected] >> >> >> >> >> > <mailto:[email protected] >> >> >> <mailto:[email protected]> >> >> >> <mailto:[email protected] <mailto:[email protected] >> >>> >> >> >> > http://blog.nanthrax.net >> >> >> > Talend - http://www.talend.com >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > Jean-Baptiste Onofré >> >> >> > [email protected] <mailto:[email protected]> >> >> >> <mailto:[email protected] <mailto:[email protected] >> >> >> >> >> > http://blog.nanthrax.net >> >> >> > Talend - http://www.talend.com >> >> >> > >> >> >> > >> >> >> >> >> >> -- >> >> >> Jean-Baptiste Onofré >> >> >> [email protected] <mailto:[email protected]> >> >> >> http://blog.nanthrax.net >> >> >> Talend - http://www.talend.com >> >> >> >> >> >> >> >> >> >> >> > >> >> > -- >> >> > Jean-Baptiste Onofré >> >> > [email protected] >> >> > http://blog.nanthrax.net >> >> > Talend - http://www.talend.com >> > >
