On Thu, Dec 7, 2017 at 3:51 PM, Eugene Kirpichov <kirpic...@google.com> wrote:
> I've sent the poll https://lists.apache.org/thread.html/ > 5bc2e184a24de9dbc8184ffd2720d1894010497d47d956b395e037df@% > 3Cuser.beam.apache.org%3E > Will figure out how to tweet from @ApacheBeam, and sent the Twitter poll > as well (or ask someone to). > I tweeted the poll. > > On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <lc...@google.com> wrote: > >> +1 on moving forward with the plan suggested by kirpichov@ >> >> On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <rober...@google.com> >> wrote: >> >>> +1 to moving forward with this plan. >>> >>> (FWIW, this seems *less* backwards incompatible than, say, moving from >>> Spark 1 to Spark 2, which was decided much quicker. I suppose the >>> Spark change has a lower bound on the number of users it could impact >>> though.) >>> >>> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <kirpic...@google.com> >>> wrote: >>> > Okay, then let's go forward. Seems that we should: >>> > - Open a new poll on user@, in light of 2.2 having been released >>> > - Open a twitter poll >>> > - Tweet that there's also a poll going on on user@ >>> > - Runner authors will reach out to respective runner user communities >>> > - 2 weeks later we gather results and decide >>> > ? >>> > >>> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <ieme...@gmail.com> wrote: >>> >> >>> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away, >>> >> and starting to improve Beam to offer a Java 8 friendly experience >>> >> seems like an excellent idea. >>> >> >>> >> I understand the backwards compatibility argument. We should do the >>> >> poll in twitter + try to reach more users for comments. If you >>> >> consider that it is worth, I can open a second poll at user@. >>> >> >>> >> In any case we should try to move forward, even if we have more than >>> >> 5% of users who want to stay on Java 7 we can consider to maintain >>> >> minor releases of a backwards compatible version where we can backport >>> >> only critical fixes e.g. security/data related errors but nothing new, >>> >> in case some user really needs to have them. Of course this can be >>> >> some extra work (to be discussed). >>> >> >>> >> >>> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré <j...@nanthrax.net >>> > >>> >> wrote: >>> >> > +1, and sorry again, I thought we got an consensus. >>> >> > >>> >> > Regards >>> >> > JB >>> >> > >>> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote: >>> >> >> >>> >> >> +1 to the poll and also to Reuven's point. >>> >> >> >>> >> >> Those without a support contract would have been using JDK 7 >>> without >>> >> >> security updates for years. IMO it seems harmful, as a netizen, to >>> >> >> encourage >>> >> >> its use/existence. >>> >> >> >>> >> >> If there's no noise from the prior thread, then I would assume no >>> one >>> >> >> on >>> >> >> user@ has any objection. Anyone else with customers should reach >>> out to >>> >> >> them. >>> >> >> >>> >> >> Kenn >>> >> >> >>> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <re...@google.com >>> >> >> <mailto:re...@google.com>> wrote: >>> >> >> >>> >> >> Technically it's a backwards-incompatible change, however if >>> we are >>> >> >> convinced the risk is low we could do it. >>> >> >> >>> >> >> As mentioned on the original thread, it's not clear that all >>> Beam >>> >> >> users read >>> >> >> user@ - e.g. most Dataflow users definitely do not. I think >>> we need >>> >> >> to >>> >> >> separately reach out to users of each runner through >>> >> >> runner-specific >>> >> >> channels. >>> >> >> >>> >> >> Reuven >>> >> >> >>> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov >>> >> >> <kirpic...@google.com >>> >> >> <mailto:kirpic...@google.com>> wrote: >>> >> >> >>> >> >> On the original thread >>> >> >> >>> >> >> >>> >> >> https://lists.apache.org/thread.html/ >>> 2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@% >>> 3Cuser.beam.apache.org%3E >>> >> >> >>> >> >> >>> >> >> <https://lists.apache.org/thread.html/ >>> 2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@% >>> 3Cuser.beam.apache.org%3E> >>> >> >> , >>> >> >> Robert and Ismaël were in favor of no major version change >>> >> >> [Ismaël >>> >> >> said:/Also I am afraid that if we wait/ >>> >> >> /until we have enough changes to switch Beam to a new major >>> >> >> version the >>> >> >> switch to Java 8 will happen too late, probably after Java >>> 8's >>> >> >> end >>> >> >> of >>> >> >> life. And I am not exaggerating, Java 8 is planned to EOL >>> next >>> >> >> march >>> >> >> 2018!/]; JB and now Reuven are in favor of a major version >>> >> >> change; >>> >> >> >>> >> >> nobody so far argued against switching to Java8 in general. >>> >> >> >>> >> >> I'm personally in favor of no major version change (i.e. >>> not >>> >> >> waiting >>> >> >> until all other large changes for Beam 3.0 converge, which >>> will >>> >> >> likely >>> >> >> be many months), because: >>> >> >> - Reasons Ismaël cited; plus the reason that most people >>> are >>> >> >> likely >>> >> >> already using Java 8. >>> >> >> - Going Java8-only earlier will make other Beam 3.0 APIs >>> better >>> >> >> for >>> >> >> Java8 users, because we (Beam contributors) will have >>> >> >> experience >>> >> >> working >>> >> >> with them within the SDK in Java8 (e.g. writing tests with >>> use >>> >> >> of >>> >> >> lambdas and noticing whether it's clunky, or whether some >>> other >>> >> >> Beam >>> >> >> APIs need better Java8 support). >>> >> >> - Going Java8 will make it more reasonable to include >>> (mostly >>> >> >> or >>> >> >> only) >>> >> >> Java8 snippets in Beam documentation, which will obviously >>> look >>> >> >> more >>> >> >> concise and attractive, addressing one of the common >>> concerns >>> >> >> of >>> >> >> Beam >>> >> >> users that it has a heavyweight API compared to >>> >> >> functional-style >>> >> >> APIs of >>> >> >> Spark etc. >>> >> >> >>> >> >> I think resolving this via a poll of users would be >>> reasonable. >>> >> >> I'd >>> >> >> suggest e.g. the following phrasing: >>> >> >> >>> >> >> Apache Beam is considering dropping support for Java 7, and >>> >> >> supporting >>> >> >> only Java 8 and above in a subsequent release. How would it >>> >> >> impact >>> >> >> your >>> >> >> usage of Beam? >>> >> >> - I am already using only Java 8+ for building my Beam code >>> >> >> - I am using Java 7 for building my Beam code, but I would >>> have >>> >> >> no >>> >> >> trouble switching to Java 8 >>> >> >> - I am using Java 7 for building my Beam code, and dropping >>> >> >> Java 7 >>> >> >> would >>> >> >> be a blocker or hindrance to adopting the new release for >>> me >>> >> >> >>> >> >> We could tweet this poll on Apache Beam twitter and >>> publish on >>> >> >> user@, >>> >> >> and, say, if we receive 5% or fewer votes for option 3 >>> after >>> >> >> keeping it >>> >> >> open for 2 weeks, then adopt Java 8 without a major version >>> >> >> change. >>> >> >> >>> >> >> WDYT? >>> >> >> >>> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré >>> >> >> <j...@nanthrax.net >>> >> >> <mailto:j...@nanthrax.net>> wrote: >>> >> >> >>> >> >> Good idea ! Definitely +1 >>> >> >> >>> >> >> Regards >>> >> >> JB >>> >> >> >>> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote: >>> >> >> > We should bring this up on the Beam 3.0 thread. >>> Since >>> >> >> it's >>> >> >> technically a >>> >> >> > backwards-incompatible change, it might make a good >>> item >>> >> >> for Beam >>> >> >> 3.0. >>> >> >> > >>> >> >> > Reuven >>> >> >> > >>> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste Onofré >>> >> >> <j...@nanthrax.net <mailto:j...@nanthrax.net> >>> >> >> > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> >>> >> >> wrote: >>> >> >> > >>> >> >> > My apologizes, I thought we had a consensus >>> already. >>> >> >> > >>> >> >> > Regards >>> >> >> > JB >>> >> >> > >>> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote: >>> >> >> > >>> >> >> > Thanks JB for sending the detailed notes >>> about >>> >> >> new >>> >> >> stuff >>> >> >> in 2.2.0! A lot >>> >> >> > of exciting things indeed. >>> >> >> > >>> >> >> > Regarding Java 8: I thought our consensus >>> was to >>> >> >> have the >>> >> >> release notes >>> >> >> > say that we're *considering* going >>> Java8-only, >>> >> >> and >>> >> >> use >>> >> >> that to get more >>> >> >> > opinions from the user community - but I >>> can't >>> >> >> find >>> >> >> the >>> >> >> emails that made >>> >> >> > me think so. >>> >> >> > >>> >> >> > +Ismaël Mejía <mailto:ieme...@gmail.com >>> >> >> <mailto:ieme...@gmail.com> <mailto:ieme...@gmail.com >>> >> >> <mailto:ieme...@gmail.com>>> - do >>> >> >> > you think we should formally conclude the >>> vote >>> >> >> on >>> >> >> the >>> >> >> thread [VOTE] >>> >> >> > [DISCUSSION] Remove support for Java 7? >>> >> >> > Or should we take more steps - e.g. perhaps >>> >> >> tweet a >>> >> >> link >>> >> >> to that thread >>> >> >> > from the Beam twitter account, ask people to >>> >> >> chime >>> >> >> in, >>> >> >> and wait for say >>> >> >> > 2 weeks before declaring a conclusion? >>> >> >> > >>> >> >> > Let's also have a process JIRA for going >>> Java8. >>> >> >> I've >>> >> >> filed one: >>> >> >> > https://issues.apache.org/jira/browse/BEAM-3285 >>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285> >>> >> >> > <https://issues.apache.org/ >>> jira/browse/BEAM-3285 >>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>> >>> >> >> > >>> >> >> > On Mon, Dec 4, 2017 at 1:58 AM Jean-Baptiste >>> >> >> Onofré >>> >> >> <j...@nanthrax.net <mailto:j...@nanthrax.net> >>> >> >> > <mailto:j...@nanthrax.net >>> >> >> <mailto:j...@nanthrax.net>> >>> >> >> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net> >>> >> >> > <mailto:j...@nanthrax.net >>> >> >> <mailto:j...@nanthrax.net>>>> >>> >> >> wrote: >>> >> >> > >>> >> >> > Just an important note that we forgot >>> to >>> >> >> mention. >>> >> >> > >>> >> >> > !! The 2.2.0 release will be the last >>> one >>> >> >> supporting >>> >> >> Spark 1.x and >>> >> >> > Java 7 !! >>> >> >> > >>> >> >> > Starting from Beam 2.3.0, the Spark >>> runner >>> >> >> will work >>> >> >> only with >>> >> >> > Spark 2.x and we >>> >> >> > will focus only Java 8. >>> >> >> > >>> >> >> > Regards >>> >> >> > JB >>> >> >> > >>> >> >> > On 12/04/2017 10:15 AM, Jean-Baptiste >>> >> >> Onofré >>> >> >> wrote: >>> >> >> > > Thanks Reuven ! >>> >> >> > > >>> >> >> > > I would like to emphasize on some >>> >> >> highlights in >>> >> >> 2.2.0 release: >>> >> >> > > >>> >> >> > > - New IOs have been introduced: >>> >> >> > > * TikaIO leveraging Apache Tika, >>> >> >> allowing >>> >> >> the >>> >> >> deal with a lot >>> >> >> > of different >>> >> >> > > data formats >>> >> >> > > * RedisIO to read and write >>> key/value >>> >> >> pairs >>> >> >> from a Redis >>> >> >> > server. This >>> >> >> > IO will >>> >> >> > > be soon extended to Redis PubSub. >>> >> >> > > * FileIO provides transforms for >>> >> >> working >>> >> >> with >>> >> >> files (raw). >>> >> >> > Especially, it >>> >> >> > > provides matching file patterns and >>> read >>> >> >> on >>> >> >> patterns. It can be >>> >> >> > easily >>> >> >> > extended >>> >> >> > > for a specific format (like we do in >>> >> >> AvroIO >>> >> >> or >>> >> >> TextIO now). >>> >> >> > > * SolrIO to interact with Apache >>> Solr >>> >> >> (Lucene) >>> >> >> > > >>> >> >> > > - On the other hand, improvements >>> have >>> >> >> been >>> >> >> performed on >>> >> >> > existing IOs: >>> >> >> > > * We started to introduce readAll >>> >> >> pattern >>> >> >> in >>> >> >> IOs (AvroIO, >>> >> >> > TextIO, JdbcIO, >>> >> >> > > ...), allowing to pass "request" >>> >> >> arguments >>> >> >> via an >>> >> >> input PCollection. >>> >> >> > > * ElasticsearchIO has an improved >>> >> >> support >>> >> >> of >>> >> >> different >>> >> >> > Elasticsearch >>> >> >> > version >>> >> >> > > (including Elasticsearch 5.x). It >>> also >>> >> >> now >>> >> >> supports SSL/TLS. >>> >> >> > > * HBaseIO is now able to do >>> dynamic >>> >> >> work >>> >> >> rebalancing >>> >> >> > > * KinesisIO uses a more accurate >>> >> >> watermark >>> >> >> (based on >>> >> >> > approximateArrivalTimestamp) >>> >> >> > > * TextIO now supports custom >>> delimiter >>> >> >> and like >>> >> >> AvroIO, >>> >> >> > supports the >>> >> >> > readAll >>> >> >> > > pattern, >>> >> >> > > * Performance improvements on >>> JdbcIO >>> >> >> when >>> >> >> it >>> >> >> has to read lot >>> >> >> > of rows >>> >> >> > > * Kafka write supports >>> Exactly-Once >>> >> >> pattern >>> >> >> (introduce in >>> >> >> > Kafka 0.11.x) >>> >> >> > > >>> >> >> > > - A new DSL has been introduced: >>> the SQL >>> >> >> DSL ! >>> >> >> > > >>> >> >> > > We are now focus on 2.3.0 release >>> with >>> >> >> new >>> >> >> improvements and >>> >> >> > features ! >>> >> >> > > >>> >> >> > > Stay tuned ! >>> >> >> > > >>> >> >> > > JB on behalf of the Apache Beam >>> >> >> community. >>> >> >> > > >>> >> >> > > On 12/02/2017 11:40 PM, Reuven Lax >>> >> >> wrote: >>> >> >> > >> The Apache Beam community is >>> pleased to >>> >> >> announce the >>> >> >> > availability of the >>> >> >> > >> 2.2.0 release. >>> >> >> > >> >>> >> >> > >> This release adds support for >>> generic >>> >> >> file >>> >> >> sources and sinks >>> >> >> > (beyond TextIO >>> >> >> > >> and AvroIO) using FileIO, including >>> >> >> support for >>> >> >> dynamic >>> >> >> > filenames using >>> >> >> > >> readAll; this allows streaming >>> >> >> pipelines >>> >> >> to now >>> >> >> read from files by >>> >> >> > >> continuously monitoring a >>> directory for >>> >> >> new >>> >> >> filw. Many other >>> >> >> > IOs are >>> >> >> > improved, >>> >> >> > >> notably including exactly-once >>> support >>> >> >> for >>> >> >> the >>> >> >> Kafka sink. Initial >>> >> >> > support for >>> >> >> > >> BEAM-SQL is also included in this >>> >> >> release. >>> >> >> For a >>> >> >> more-complete >>> >> >> > list of major >>> >> >> > >> changes in the release, please >>> refer to >>> >> >> the >>> >> >> release notes [2]. >>> >> >> > >> >>> >> >> > >> The 2.2.0 release is now the >>> >> >> recommended >>> >> >> version; we encourage >>> >> >> > everyone to >>> >> >> > >> upgrade from any earlier releases. >>> >> >> > >> >>> >> >> > >> We’d like to invite everyone to >>> try out >>> >> >> Apache >>> >> >> Beam today and >>> >> >> > consider >>> >> >> > >> joining our vibrant community. We >>> >> >> welcome >>> >> >> feedback, >>> >> >> > contribution and >>> >> >> > >> participation through our mailing >>> >> >> lists, >>> >> >> issue >>> >> >> tracker, pull >>> >> >> > requests, and >>> >> >> > >> events. >>> >> >> > >> >>> >> >> > >> - Reuven Lax, on behalf of the >>> Apache >>> >> >> Beam >>> >> >> community. >>> >> >> > >> >>> >> >> > >> [1] >>> >> >> https://beam.apache.org/get-started/downloads/ >>> >> >> <https://beam.apache.org/get-started/downloads/> >>> >> >> > <https://beam.apache.org/get- >>> started/downloads/ >>> >> >> <https://beam.apache.org/get-started/downloads/>> >>> >> >> > >> [2] >>> >> >> > >> >>> >> >> > >>> >> >> >>> >> >> >>> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa? >>> projectId=12319527&version=12341044 >>> >> >> >>> >> >> >>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? >>> projectId=12319527&version=12341044> >>> >> >> > >>> >> >> >>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? >>> projectId=12319527&version=12341044 >>> >> >> >>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? >>> projectId=12319527&version=12341044>> >>> >> >> > >> >>> >> >> > > >>> >> >> > >>> >> >> > -- >>> >> >> > Jean-Baptiste Onofré >>> >> >> > jbono...@apache.org <mailto:jbono...@apache.org> >>> >> >> <mailto:jbono...@apache.org <mailto: >>> jbono...@apache.org>> >>> >> >> > <mailto:jbono...@apache.org >>> >> >> <mailto:jbono...@apache.org> >>> >> >> <mailto:jbono...@apache.org <mailto: >>> jbono...@apache.org>>> >>> >> >> > http://blog.nanthrax.net >>> >> >> > Talend - http://www.talend.com >>> >> >> > >>> >> >> > >>> >> >> > -- >>> >> >> > Jean-Baptiste Onofré >>> >> >> > jbono...@apache.org <mailto:jbono...@apache.org> >>> >> >> <mailto:jbono...@apache.org <mailto: >>> jbono...@apache.org>> >>> >> >> > http://blog.nanthrax.net >>> >> >> > Talend - http://www.talend.com >>> >> >> > >>> >> >> > >>> >> >> >>> >> >> -- >>> >> >> Jean-Baptiste Onofré >>> >> >> jbono...@apache.org <mailto:jbono...@apache.org> >>> >> >> http://blog.nanthrax.net >>> >> >> Talend - http://www.talend.com >>> >> >> >>> >> >> >>> >> >> >>> >> > >>> >> > -- >>> >> > Jean-Baptiste Onofré >>> >> > jbono...@apache.org >>> >> > http://blog.nanthrax.net >>> >> > Talend - http://www.talend.com >>> >> >>
