Hi all, due to issues found by Udi and Pablo, I cancel RC1 to prepare a new release.
Thanks Udi and Pablo for catching up. I will prepare a RC2 after cherry pick/fixed the found issues (tomorrow morning my time). Regards JB On 12/06/2018 21:04, Pablo Estrada wrote: > Hello all, > I've found another release blocker: The current findbugs dependency in > build_rules.gradle has a GPL license. Here's the PR to depend on the > correct findbugs: https://github.com/apache/beam/pull/5609 > > Best > -P. > > On Tue, Jun 12, 2018 at 11:03 AM Udi Meiri <[email protected] > <mailto:[email protected]>> wrote: > > Cherrypick created: https://github.com/apache/beam/pull/5607 > Tests still running. > > On Tue, Jun 12, 2018 at 9:59 AM Udi Meiri <[email protected] > <mailto:[email protected]>> wrote: > > -1: Would like to cherry pick a fix > for https://issues.apache.org/jira/browse/BEAM-4536 > > On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré > <[email protected] <mailto:[email protected]>> wrote: > > Any update about your vote and fix ? > > Thanks ! > Regards > JB > > On 12/06/2018 04:02, Udi Meiri wrote: > > Another bug: reading from PubSub with_attributes=True is > broken on > > Python with Dataflow. > > https://issues.apache.org/jira/browse/BEAM-4536 > > > > JB, I'm making a PR that removes this keyword and I'd like > to propose it > > as a cherrypick to 2.5.0. > > (feature should be fixed in the next release) > > > > On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>> wrote: > > > > FYI: looks like Python tests are failing for Windows. JIRA > > is https://issues.apache.org/jira/browse/BEAM-4535. > > > > I don't think this is a release blocker but this > should probably go > > in release notes (for any user that tries to run tests > on Python > > source build). And we should try to incorporate a fix > if we happen > > to cut another release candidate for some reason. > > > > Thanks, > > Cham > > > > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>> wrote: > > > > Thanks everyone who has pitched in to validate the > release! > > > > Boyuan Zhang and I have also run a few pipelines, > and verified > > that they work properly (see release validation > spreadsheet[1]). > > > > We have also found that the Game Stats pipeline is > failing in > > Python Streaming Dataflow. I have filed > BEAM-4534[2]. This is > > not a blocker, since Python streaming is not yet > fully supported. > > > > It seems that the uploaded artifacts look good. > > > > We have noticed that the Python artifacts are > still missing > > Python wheel files (compare [3] and [4]). JB, > could you please > > add the wheel files? Boyuan and I can try to help > you prepare > > them / upload them if necessary. Please let us know. > > > > Thanks again! > > -P. > > > > > [1] > https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807 > > [2] https://issues.apache.org/jira/browse/BEAM-4534 > > [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/ > > [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/ > > > > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold > > <[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> > wrote: > > > > +1 (non-binding) > > > > tested some of the quickstarts > > > > On Sun, Jun 10, 2018 at 1:39 AM Tim > > <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>> wrote: > > > > Tested by our team: > > - mvn inclusion > > - Avro, ES, Hadoop IF IO > > - Pipelines run on Spark (Cloudera 5.12.0 > YARN cluster) > > - Reviewed release notes > > > > +1 > > > > Thanks also to everyone who helped get > over the gradle > > hurdle and in particular to JB. > > > > Tim > > > > > On 9 Jun 2018, at 05:56, Jean-Baptiste > Onofré > > <[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > > > No problem Pablo. > > > > > > The vote period is a minimum, it can be > extended as > > requested or if we > > > don't have the minimum of 3 binding votes. > > > > > > Regards > > > JB > > > > > >> On 09/06/2018 01:54, Pablo Estrada wrote: > > >> Hello all, > > >> I'd like to request an extension of the > voting period > > until Monday > > >> evening (US time, so later in other > geographical > > regions). This is > > >> because we were only now able to > publish Dataflow > > Workers, and have not > > >> had the chance to run release > validation tests on > > them. The extension > > >> will allow us to validate and vote by > Monday. > > >> Is this acceptable to the community? > > >> > > >> Best > > >> -P. > > >> > > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey > Romanenko > > >> <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > <mailto:[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>>> wrote: > > >> > > >> Thank you JB for your work! > > >> > > >> I tested running simple streaming > (/KafkaIO/) and > > batch (/TextIO / > > >> HDFS/) pipelines with SparkRunner on > YARN cluster > > - it works fine. > > >> > > >> WBR, > > >> Alexey > > >> > > >> > > >>> On 8 Jun 2018, at 10:00, Etienne > Chauchot > > <[email protected] > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>> > > >>> <mailto:[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>>> wrote: > > >>> > > >>> I forgot to vote: > > >>> +1 (non binding). > > >>> What I tested: > > >>> - no functional or performance > regression > > comparing to v2.4 > > >>> - dependencies in the poms are ok > > >>> > > >>> Etienne > > >>>> Le vendredi 08 juin 2018 à 08:27 > +0200, Romain > > Manni-Bucau a écrit : > > >>>> +1 (non-binding), mainstream usage > is not broken > > by the pom > > >>>> changes and runtime has no known > regression > > compared to the 2.4.0 > > >>>> > > >>>> (side note: kudo to JB for this > build tool > > change release, I know > > >>>> how it can hurt ;)) > > >>>> > > >>>> Romain Manni-Bucau > > >>>> @rmannibucau > <https://twitter.com/rmannibucau> > > | Blog > > >>>> > <https://rmannibucau.metawerx.net/> | Old Blog > > >>>> > <http://rmannibucau.wordpress.com/> | Github > > >>>> <https://github.com/rmannibucau> | > LinkedIn > > >>>> > <https://www.linkedin.com/in/rmannibucau> | Book > > >>>> > > > > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > >>>> > > >>>> > > >>>> Le jeu. 7 juin 2018 à 16:17, > Jean-Baptiste Onofré > > >>>> <[email protected] > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>> > > <mailto:[email protected] > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>>>> a écrit : > > >>>>> Thanks for the details Etienne ! > > >>>>> > > >>>>> The good news is that the > artifacts seem OK and > > the overall Nexmark > > >>>>> results are consistent with the > 2.4.0 release ones. > > >>>>> > > >>>>> I'm starting a complete review > using the > > beam-samples as well. > > >>>>> > > >>>>> Regards > > >>>>> JB > > >>>>> > > >>>>>> On 07/06/2018 16:14, Etienne > Chauchot wrote: > > >>>>>> Hi, > > >>>>>> I've just run the nexmark queries > on v2.5.0-RC1 tag > > >>>>>> What we can notice: > > >>>>>> - query 3 (exercises CoGroupByKey, > state and > > timer) shows > > >>>>> different > > >>>>>> output with DR between batch and > streaming and > > with the other > > >>>>> runners => > > >>>>>> I compared with v2.4 there were > still these > > differences but with > > >>>>>> different output size numbers > > >>>>>> > > >>>>>> - query 6 (exercises specialized > combiner) shows > > different output > > >>>>>> between the runners => the correct > output is 401. > > strange that > > >>>>> in batch > > >>>>>> mode some runners output les > Sellers. I compared > > with v2.4 > > >>>>> same output > > >>>>>> > > >>>>>> - response time of query 7 > (exercices Max > > transform, fanout > > >>>>> and side > > >>>>>> input) is very slow on DR => I > compared with v2.4 > > , comparable > > >>>>> execution > > >>>>>> times > > >>>>>> > > >>>>>> I'm not comparing q10 because it is > a write to > > GCS so it is > > >>>>> very specific. > > >>>>>> > > >>>>>> => Basically no regression > comparing to v2.4 > > >>>>>> > > >>>>>> For the record here is the output > (waiting for > > ongoing perfkit > > >>>>> integration): > > >>>>>> > > >>>>>> > > >>>>>> 1. DR batch > > >>>>>> > > >>>>>> Performance: > > >>>>>> > > >>>>>> > > >>>>> Conf Runtime(sec) (Baseline) > Events(/sec) > > (Baseline) Results (Baseline) > > >>>>>> > > >>>>>> > > >>>>> 0000 5,8 > 17283,1 > > 100000 > > >>>>>> > > >>>>>> > > >>>>> 0001 3,2 > 31104,2 > > 92000 > > >>>>>> > > >>>>>> > > >>>>> 0002 1,2 > 82918,7 > > 351 > > >>>>>> > > >>>>>> > > >>>>> 0003 2,2 > 46210,7 > > 458 > > >>>>>> > > >>>>>> > > >>>>> 0004 1,2 > 8503,4 > > 40 > > >>>>>> > > >>>>>> > > >>>>> 0005 4,0 > 25220,7 > > 12 > > >>>>>> > > >>>>>> > > >>>>> 0006 0,9 > 11148,3 > > 401 > > >>>>>> > > >>>>>> > > >>>>> 0007 13,2 > 7580,9 > > 1 > > >>>>>> > > >>>>>> > > >>>>> 0008 1,5 > 67340,1 > > 6000 > > >>>>>> > > >>>>>> > > >>>>> 0009 0,7 > 14025,2 > > 298 > > >>>>>> > > >>>>>> > > >>>>> 0010 12,8 > 7793,0 > > 1 > > >>>>>> > > >>>>>> > > >>>>> 0011 2,4 > 42319,1 > > 1919 > > >>>>>> > > >>>>>> > > >>>>> 0012 1,6 > 61462,8 > > 1919 > > >>>>>> > > >>>>> > > > > ========================================================================================== > > >>>>>> > > >>>>>> 2. DR streaming > > >>>>>> > > >>>>>> Performance: > > >>>>>> > > >>>>>> > > >>>>> Conf Runtime(sec) (Baseline) > Events(/sec) > > (Baseline) Results (Baseline) > > >>>>>> > > >>>>>> > > >>>>> 0000 6,5 > 15285,8 > > 100000 > > >>>>>> > > >>>>>> > > >>>>> 0001 3,7 > 27397,3 > > 92000 > > >>>>>> > > >>>>>> > > >>>>> 0002 1,4 > 69108,5 > > 351 > > >>>>>> > > >>>>>> > > >>>>> 0003 3,2 > 31181,8 > > 447 > > >>>>>> > > >>>>>> > > >>>>> 0004 1,2 > 8361,2 > > 40 > > >>>>>> > > >>>>>> > > >>>>> 0005 5,3 > 18903,6 > > 12 > > >>>>>> > > >>>>>> > > >>>>> 0006 0,9 > 11111,1 > > 401 > > >>>>>> > > >>>>>> > > >>>>> 0007 82,5 > 1212,2 > > 1 > > >>>>>> > > >>>>>> > > >>>>> 0008 2,0 > 51072,5 > > 6000 > > >>>>>> > > >>>>>> > > >>>>> 0009 0,8 > 12903,2 > > 298 > > >>>>>> > > >>>>>> > > >>>>> 0010 49,5 > 2021,8 > > 1 > > >>>>>> > > >>>>>> > > >>>>> 0011 3,9 > 25667,4 > > 1919 > > >>>>>> > > >>>>>> > > >>>>> 0012 2,4 > 41067,8 > > 1919 > > >>>>>> > > >>>>> > > > > ========================================================================================== > > >>>>>> > > >>>>>> 3. Flink batch > > >>>>>> Performance: > > >>>>>> > > >>>>>> > > >>>>> Conf Runtime(sec) (Baseline) > Events(/sec) > > (Baseline) Results (Baseline) > > >>>>>> > > >>>>>> > > >>>>> 0000 1,0 > 97656,3 > > 100000 > > >>>>>> > > >>>>>> > > >>>>> 0001 0,7 > 141643,1 > > 92000 > > >>>>>> > > >>>>>> > > >>>>> 0002 0,4 > 228310,5 > > 351 > > >>>>>> > > >>>>>> > > >>>>> 0003 1,6 > 64020,5 > > 580 > > >>>>>> > > >>>>>> > > >>>>> 0004 0,7 > 13831,3 > > 40 > > >>>>>> > > >>>>>> > > >>>>> 0005 1,4 > 72939,5 > > 12 > > >>>>>> > > >>>>>> > > >>>>> 0006 0,5 > 20491,8 > > 103 > > >>>>>> > > >>>>>> > > >>>>> 0007 1,3 > 74239,0 > > 1 > > >>>>>> > > >>>>>> > > >>>>> 0008 0,8 > 121506,7 > > 6000 > > >>>>>> > > >>>>>> > > >>>>> 0009 0,6 > 17953,3 > > 298 > > >>>>>> > > >>>>>> > > >>>>> 0010 1,3 > 74682,6 > > 1 > > >>>>>> > > >>>>>> > > >>>>> 0011 1,1 > 92936,8 > > 1919 > > >>>>>> > > >>>>>> > > >>>>> 0012 0,8 > 123001,2 > > 1919 > > >>>>>> > > >>>>> > > > > ========================================================================================== > > >>>>>> > > >>>>>> 4. Flink streaming > > >>>>>> Performance: > > >>>>>> > > >>>>>> > > >>>>> Conf Runtime(sec) (Baseline) > Events(/sec) > > (Baseline) Results (Baseline) > > >>>>>> > > >>>>>> > > >>>>> 0000 5,4 > 18677,6 > > 100000 > > >>>>>> > > >>>>>> > > >>>>> 0001 2,8 > 35511,4 > > 92000 > > >>>>>> > > >>>>>> > > >>>>> 0002 1,8 > 54318,3 > > 351 > > >>>>>> > > >>>>>> > > >>>>> 0003 2,4 > 41614,6 > > 580 > > >>>>>> > > >>>>>> > > >>>>> 0004 1,0 > 10341,3 > > 40 > > >>>>>> > > >>>>>> > > >>>>> 0005 3,4 > 29568,3 > > 12 > > >>>>>> > > >>>>>> > > >>>>> 0006 0,7 > 13369,0 > > 401 > > >>>>>> > > >>>>>> > > >>>>> 0007 2,8 > 36192,5 > > 1 > > >>>>>> > > >>>>>> > > >>>>> 0008 1,8 > 54854,6 > > 6000 > > >>>>>> > > >>>>>> > > >>>>> 0009 0,7 > 13369,0 > > 298 > > >>>>>> > > >>>>>> > > >>>>> 0010 3,4 > 29841,8 > > 2 > > >>>>>> > > >>>>>> > > >>>>> 0011 5,0 > 19932,2 > > 1919 > > >>>>>> > > >>>>>> > > >>>>> 0012 2,6 > 38835,0 > > 1919 > > >>>>>> > > >>>>> > > > > ========================================================================================== > > >>>>>> > > >>>>>> 5. Spark batch > > >>>>>> Performance: > > >>>>>> > > >>>>>> > > >>>>> Conf Runtime(sec) (Baseline) > Events(/sec) > > (Baseline) Results (Baseline) > > >>>>>> > > >>>>>> > > >>>>> 0000 1,5 > 65445,0 > > 100000 > > >>>>>> > > >>>>>> > > >>>>> 0001 1,3 > 79491,3 > > 92000 > > >>>>>> > > >>>>>> > > >>>>> 0002 0,9 > 112107,6 > > 351 > > >>>>>> > > >>>>>> > > >>>>> 0003 2,0 > 48804,3 > > 580 > > >>>>>> > > >>>>>> > > >>>>> 0004 1,2 > 8382,2 > > 40 > > >>>>>> > > >>>>>> > > >>>>> 0005 2,0 > 50838,8 > > 12 > > >>>>>> > > >>>>>> > > >>>>> 0006 1,0 > 9699,3 > > 103 > > >>>>>> > > >>>>>> > > >>>>> 0007 2,3 > 43308,8 > > 1 > > >>>>>> > > >>>>>> > > >>>>> 0008 2,1 > 46794,6 > > 6000 > > >>>>>> > > >>>>>> > > >>>>> 0009 1,1 > 8976,7 > > 298 > > >>>>>> > > >>>>>> > > >>>>> 0010 1,6 > 62111,8 > > 1 > > >>>>>> > > >>>>>> > > >>>>> 0011 2,1 > 46598,3 > > 1919 > > >>>>>> > > >>>>>> > > >>>>> 0012 2,3 > 43687,2 > > 1919 > > >>>>>> > > >>>>> > > > > ========================================================================================== > > >>>>>> > > >>>>>>> Le mercredi 06 juin 2018 à 10:50 > +0200, Etienne > > Chauchot a écrit : > > >>>>>>> Thanks JB for all your work ! I > believe doing > > the first > > >>>>> gradle release > > >>>>>>> must have been hard. > > >>>>>>> I'll run Nexmark on the release > and keep you posted. > > >>>>>>> > > >>>>>>> Best > > >>>>>>> Etienne > > >>>>>>> > > >>>>>>> > > >>>>>>> Le mercredi 06 juin 2018 à 10:44 > +0200, > > Jean-Baptiste Onofré > > >>>>> a écrit : > > >>>>>>>> Hi everyone, > > >>>>>>>> > > >>>>>>>> Please review and vote on the > release candidate > > #1 for the > > >>>>> version > > >>>>>>>> 2.5.0, as follows: > > >>>>>>>> > > >>>>>>>> [ ] +1, Approve the release > > >>>>>>>> [ ] -1, Do not approve the > release (please > > provide specific > > >>>>> comments) > > >>>>>>>> > > >>>>>>>> NB: this is the first release > using Gradle, so > > don't be too > > >>>>> harsh ;) A > > >>>>>>>> PR about the release guide will > follow thanks > > to this release. > > >>>>>>>> > > >>>>>>>> The complete staging area is > available for your > > review, > > >>>>> which includes: > > >>>>>>>> * JIRA release notes [1], > > >>>>>>>> * the official Apache source > release to be > > deployed to > > >>>>> dist.apache.org > <http://dist.apache.org> <http://dist.apache.org> > > <http://dist.apache.org/> > > >>>>>>>> [2], which is signed with the key > with > > fingerprint C8282E76 [3], > > >>>>>>>> * all artifacts to be deployed to > the Maven Central > > >>>>> Repository [4], > > >>>>>>>> * source code tag "v2.5.0-RC1" [5], > > >>>>>>>> * website pull request listing > the release and > > publishing > > >>>>> the API > > >>>>>>>> reference manual [6]. > > >>>>>>>> * Java artifacts were built with > Gradle 4.7 > > (wrapper) and > > >>>>> OpenJDK/Oracle > > >>>>>>>> JDK 1.8.0_172 (Oracle Corporation > 25.172-b11). > > >>>>>>>> * Python artifacts are deployed > along with the > > source > > >>>>> release to the > > >>>>>>>> dist.apache.org > <http://dist.apache.org> <http://dist.apache.org> > > <http://dist.apache.org/> [2]. > > >>>>>>>> > > >>>>>>>> The vote will be open for at > least 72 hours. It > > is adopted > > >>>>> by majority > > >>>>>>>> approval, with at least 3 PMC > affirmative votes. > > >>>>>>>> > > >>>>>>>> Thanks, > > >>>>>>>> JB > > >>>>>>>> > > >>>>>>>> [1] > > >>>>>>>> > > >>>>> > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847 > > >>>>>>>> [2] > > > https://dist.apache.org/repos/dist/dev/beam/2.5.0/ > > >>>>>>>> [3] > > > https://dist.apache.org/repos/dist/release/beam/KEYS > > >>>>>>>> [4] > > >>>>> > > > > https://repository.apache.org/content/repositories/orgapachebeam-1041/ > > >>>>>>>> [5] > https://github.com/apache/beam/tree/v2.5.0-RC1 > > >>>>>>>> [6] > https://github.com/apache/beam-site/pull/463 > > >>>>>>>> > > >>>>> > > >> > > >> -- > > >> Got feedback? go/pabloem-feedback > <https://goto.google.com/pabloem-feedback> > > <https://goto.google.com/pabloem-feedback> > > > > > > -- > > > Jean-Baptiste Onofré > > > [email protected] > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>> > > > http://blog.nanthrax.net > > > Talend - http://www.talend.com > > > > -- > > Got feedback? go/pabloem-feedback > <https://goto.google.com/pabloem-feedback> > > <https://goto.google.com/pabloem-feedback> > > > > -- > Jean-Baptiste Onofré > [email protected] <mailto:[email protected]> > http://blog.nanthrax.net > Talend - http://www.talend.com > > -- > Got feedback? go/pabloem-feedback
