Thanks everyone who has pitched in to validate the release! Boyuan Zhang and I have also run a few pipelines, and verified that they work properly (see release validation spreadsheet[1]).
We have also found that the Game Stats pipeline is failing in Python Streaming Dataflow. I have filed BEAM-4534[2]. This is not a blocker, since Python streaming is not yet fully supported. It seems that the uploaded artifacts look good. We have noticed that the Python artifacts are still missing Python wheel files (compare [3] and [4]). JB, could you please add the wheel files? Boyuan and I can try to help you prepare them / upload them if necessary. Please let us know. Thanks again! -P. [1] https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807 [2] https://issues.apache.org/jira/browse/BEAM-4534 [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/ [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/ On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold <amyrv...@google.com> wrote: > +1 (non-binding) > > tested some of the quickstarts > > On Sun, Jun 10, 2018 at 1:39 AM Tim <timrobertson...@gmail.com> wrote: > >> Tested by our team: >> - mvn inclusion >> - Avro, ES, Hadoop IF IO >> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster) >> - Reviewed release notes >> >> +1 >> >> Thanks also to everyone who helped get over the gradle hurdle and in >> particular to JB. >> >> Tim >> >> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: >> > >> > No problem Pablo. >> > >> > The vote period is a minimum, it can be extended as requested or if we >> > don't have the minimum of 3 binding votes. >> > >> > Regards >> > JB >> > >> >> On 09/06/2018 01:54, Pablo Estrada wrote: >> >> Hello all, >> >> I'd like to request an extension of the voting period until Monday >> >> evening (US time, so later in other geographical regions). This is >> >> because we were only now able to publish Dataflow Workers, and have not >> >> had the chance to run release validation tests on them. The extension >> >> will allow us to validate and vote by Monday. >> >> Is this acceptable to the community? >> >> >> >> Best >> >> -P. >> >> >> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko >> >> <aromanenko....@gmail.com <mailto:aromanenko....@gmail.com>> wrote: >> >> >> >> Thank you JB for your work! >> >> >> >> I tested running simple streaming (/KafkaIO/) and batch (/TextIO / >> >> HDFS/) pipelines with SparkRunner on YARN cluster - it works fine. >> >> >> >> WBR, >> >> Alexey >> >> >> >> >> >>> On 8 Jun 2018, at 10:00, Etienne Chauchot <echauc...@apache.org >> >>> <mailto:echauc...@apache.org>> wrote: >> >>> >> >>> I forgot to vote: >> >>> +1 (non binding). >> >>> What I tested: >> >>> - no functional or performance regression comparing to v2.4 >> >>> - dependencies in the poms are ok >> >>> >> >>> Etienne >> >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a >> écrit : >> >>>> +1 (non-binding), mainstream usage is not broken by the pom >> >>>> changes and runtime has no known regression compared to the 2.4.0 >> >>>> >> >>>> (side note: kudo to JB for this build tool change release, I know >> >>>> how it can hurt ;)) >> >>>> >> >>>> Romain Manni-Bucau >> >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >> >>>> <https://rmannibucau.metawerx.net/> | Old Blog >> >>>> <http://rmannibucau.wordpress.com/> | Github >> >>>> <https://github.com/rmannibucau> | LinkedIn >> >>>> <https://www.linkedin.com/in/rmannibucau> | Book >> >>>> < >> https://www.packtpub.com/application-development/java-ee-8-high-performance >> > >> >>>> >> >>>> >> >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré >> >>>> <j...@nanthrax.net <mailto:j...@nanthrax.net>> a écrit : >> >>>>> Thanks for the details Etienne ! >> >>>>> >> >>>>> The good news is that the artifacts seem OK and the overall >> Nexmark >> >>>>> results are consistent with the 2.4.0 release ones. >> >>>>> >> >>>>> I'm starting a complete review using the beam-samples as well. >> >>>>> >> >>>>> Regards >> >>>>> JB >> >>>>> >> >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote: >> >>>>>> Hi, >> >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag >> >>>>>> What we can notice: >> >>>>>> - query 3 (exercises CoGroupByKey, state and timer) shows >> >>>>> different >> >>>>>> output with DR between batch and streaming and with the other >> >>>>> runners => >> >>>>>> I compared with v2.4 there were still these differences but with >> >>>>>> different output size numbers >> >>>>>> >> >>>>>> - query 6 (exercises specialized combiner) shows different output >> >>>>>> between the runners => the correct output is 401. strange that >> >>>>> in batch >> >>>>>> mode some runners output les Sellers. I compared with v2.4 >> >>>>> same output >> >>>>>> >> >>>>>> - response time of query 7 (exercices Max transform, fanout >> >>>>> and side >> >>>>>> input) is very slow on DR => I compared with v2.4 , comparable >> >>>>> execution >> >>>>>> times >> >>>>>> >> >>>>>> I'm not comparing q10 because it is a write to GCS so it is >> >>>>> very specific. >> >>>>>> >> >>>>>> => Basically no regression comparing to v2.4 >> >>>>>> >> >>>>>> For the record here is the output (waiting for ongoing perfkit >> >>>>> integration): >> >>>>>> >> >>>>>> >> >>>>>> 1. DR batch >> >>>>>> >> >>>>>> Performance: >> >>>>>> >> >>>>>> >> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) >> Results (Baseline) >> >>>>>> >> >>>>>> >> >>>>> 0000 5,8 17283,1 >> 100000 >> >>>>>> >> >>>>>> >> >>>>> 0001 3,2 31104,2 >> 92000 >> >>>>>> >> >>>>>> >> >>>>> 0002 1,2 82918,7 >> 351 >> >>>>>> >> >>>>>> >> >>>>> 0003 2,2 46210,7 >> 458 >> >>>>>> >> >>>>>> >> >>>>> 0004 1,2 8503,4 >> 40 >> >>>>>> >> >>>>>> >> >>>>> 0005 4,0 25220,7 >> 12 >> >>>>>> >> >>>>>> >> >>>>> 0006 0,9 11148,3 >> 401 >> >>>>>> >> >>>>>> >> >>>>> 0007 13,2 7580,9 >> 1 >> >>>>>> >> >>>>>> >> >>>>> 0008 1,5 67340,1 >> 6000 >> >>>>>> >> >>>>>> >> >>>>> 0009 0,7 14025,2 >> 298 >> >>>>>> >> >>>>>> >> >>>>> 0010 12,8 7793,0 >> 1 >> >>>>>> >> >>>>>> >> >>>>> 0011 2,4 42319,1 >> 1919 >> >>>>>> >> >>>>>> >> >>>>> 0012 1,6 61462,8 >> 1919 >> >>>>>> >> >>>>> >> ========================================================================================== >> >>>>>> >> >>>>>> 2. DR streaming >> >>>>>> >> >>>>>> Performance: >> >>>>>> >> >>>>>> >> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) >> Results (Baseline) >> >>>>>> >> >>>>>> >> >>>>> 0000 6,5 15285,8 >> 100000 >> >>>>>> >> >>>>>> >> >>>>> 0001 3,7 27397,3 >> 92000 >> >>>>>> >> >>>>>> >> >>>>> 0002 1,4 69108,5 >> 351 >> >>>>>> >> >>>>>> >> >>>>> 0003 3,2 31181,8 >> 447 >> >>>>>> >> >>>>>> >> >>>>> 0004 1,2 8361,2 >> 40 >> >>>>>> >> >>>>>> >> >>>>> 0005 5,3 18903,6 >> 12 >> >>>>>> >> >>>>>> >> >>>>> 0006 0,9 11111,1 >> 401 >> >>>>>> >> >>>>>> >> >>>>> 0007 82,5 1212,2 >> 1 >> >>>>>> >> >>>>>> >> >>>>> 0008 2,0 51072,5 >> 6000 >> >>>>>> >> >>>>>> >> >>>>> 0009 0,8 12903,2 >> 298 >> >>>>>> >> >>>>>> >> >>>>> 0010 49,5 2021,8 >> 1 >> >>>>>> >> >>>>>> >> >>>>> 0011 3,9 25667,4 >> 1919 >> >>>>>> >> >>>>>> >> >>>>> 0012 2,4 41067,8 >> 1919 >> >>>>>> >> >>>>> >> ========================================================================================== >> >>>>>> >> >>>>>> 3. Flink batch >> >>>>>> Performance: >> >>>>>> >> >>>>>> >> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) >> Results (Baseline) >> >>>>>> >> >>>>>> >> >>>>> 0000 1,0 97656,3 >> 100000 >> >>>>>> >> >>>>>> >> >>>>> 0001 0,7 141643,1 >> 92000 >> >>>>>> >> >>>>>> >> >>>>> 0002 0,4 228310,5 >> 351 >> >>>>>> >> >>>>>> >> >>>>> 0003 1,6 64020,5 >> 580 >> >>>>>> >> >>>>>> >> >>>>> 0004 0,7 13831,3 >> 40 >> >>>>>> >> >>>>>> >> >>>>> 0005 1,4 72939,5 >> 12 >> >>>>>> >> >>>>>> >> >>>>> 0006 0,5 20491,8 >> 103 >> >>>>>> >> >>>>>> >> >>>>> 0007 1,3 74239,0 >> 1 >> >>>>>> >> >>>>>> >> >>>>> 0008 0,8 121506,7 >> 6000 >> >>>>>> >> >>>>>> >> >>>>> 0009 0,6 17953,3 >> 298 >> >>>>>> >> >>>>>> >> >>>>> 0010 1,3 74682,6 >> 1 >> >>>>>> >> >>>>>> >> >>>>> 0011 1,1 92936,8 >> 1919 >> >>>>>> >> >>>>>> >> >>>>> 0012 0,8 123001,2 >> 1919 >> >>>>>> >> >>>>> >> ========================================================================================== >> >>>>>> >> >>>>>> 4. Flink streaming >> >>>>>> Performance: >> >>>>>> >> >>>>>> >> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) >> Results (Baseline) >> >>>>>> >> >>>>>> >> >>>>> 0000 5,4 18677,6 >> 100000 >> >>>>>> >> >>>>>> >> >>>>> 0001 2,8 35511,4 >> 92000 >> >>>>>> >> >>>>>> >> >>>>> 0002 1,8 54318,3 >> 351 >> >>>>>> >> >>>>>> >> >>>>> 0003 2,4 41614,6 >> 580 >> >>>>>> >> >>>>>> >> >>>>> 0004 1,0 10341,3 >> 40 >> >>>>>> >> >>>>>> >> >>>>> 0005 3,4 29568,3 >> 12 >> >>>>>> >> >>>>>> >> >>>>> 0006 0,7 13369,0 >> 401 >> >>>>>> >> >>>>>> >> >>>>> 0007 2,8 36192,5 >> 1 >> >>>>>> >> >>>>>> >> >>>>> 0008 1,8 54854,6 >> 6000 >> >>>>>> >> >>>>>> >> >>>>> 0009 0,7 13369,0 >> 298 >> >>>>>> >> >>>>>> >> >>>>> 0010 3,4 29841,8 >> 2 >> >>>>>> >> >>>>>> >> >>>>> 0011 5,0 19932,2 >> 1919 >> >>>>>> >> >>>>>> >> >>>>> 0012 2,6 38835,0 >> 1919 >> >>>>>> >> >>>>> >> ========================================================================================== >> >>>>>> >> >>>>>> 5. Spark batch >> >>>>>> Performance: >> >>>>>> >> >>>>>> >> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) >> Results (Baseline) >> >>>>>> >> >>>>>> >> >>>>> 0000 1,5 65445,0 >> 100000 >> >>>>>> >> >>>>>> >> >>>>> 0001 1,3 79491,3 >> 92000 >> >>>>>> >> >>>>>> >> >>>>> 0002 0,9 112107,6 >> 351 >> >>>>>> >> >>>>>> >> >>>>> 0003 2,0 48804,3 >> 580 >> >>>>>> >> >>>>>> >> >>>>> 0004 1,2 8382,2 >> 40 >> >>>>>> >> >>>>>> >> >>>>> 0005 2,0 50838,8 >> 12 >> >>>>>> >> >>>>>> >> >>>>> 0006 1,0 9699,3 >> 103 >> >>>>>> >> >>>>>> >> >>>>> 0007 2,3 43308,8 >> 1 >> >>>>>> >> >>>>>> >> >>>>> 0008 2,1 46794,6 >> 6000 >> >>>>>> >> >>>>>> >> >>>>> 0009 1,1 8976,7 >> 298 >> >>>>>> >> >>>>>> >> >>>>> 0010 1,6 62111,8 >> 1 >> >>>>>> >> >>>>>> >> >>>>> 0011 2,1 46598,3 >> 1919 >> >>>>>> >> >>>>>> >> >>>>> 0012 2,3 43687,2 >> 1919 >> >>>>>> >> >>>>> >> ========================================================================================== >> >>>>>> >> >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit : >> >>>>>>> Thanks JB for all your work ! I believe doing the first >> >>>>> gradle release >> >>>>>>> must have been hard. >> >>>>>>> I'll run Nexmark on the release and keep you posted. >> >>>>>>> >> >>>>>>> Best >> >>>>>>> Etienne >> >>>>>>> >> >>>>>>> >> >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré >> >>>>> a écrit : >> >>>>>>>> Hi everyone, >> >>>>>>>> >> >>>>>>>> Please review and vote on the release candidate #1 for the >> >>>>> version >> >>>>>>>> 2.5.0, as follows: >> >>>>>>>> >> >>>>>>>> [ ] +1, Approve the release >> >>>>>>>> [ ] -1, Do not approve the release (please provide specific >> >>>>> comments) >> >>>>>>>> >> >>>>>>>> NB: this is the first release using Gradle, so don't be too >> >>>>> harsh ;) A >> >>>>>>>> PR about the release guide will follow thanks to this release. >> >>>>>>>> >> >>>>>>>> The complete staging area is available for your review, >> >>>>> which includes: >> >>>>>>>> * JIRA release notes [1], >> >>>>>>>> * the official Apache source release to be deployed to >> >>>>> dist.apache.org <http://dist.apache.org/> >> >>>>>>>> [2], which is signed with the key with fingerprint C8282E76 [3], >> >>>>>>>> * all artifacts to be deployed to the Maven Central >> >>>>> Repository [4], >> >>>>>>>> * source code tag "v2.5.0-RC1" [5], >> >>>>>>>> * website pull request listing the release and publishing >> >>>>> the API >> >>>>>>>> reference manual [6]. >> >>>>>>>> * Java artifacts were built with Gradle 4.7 (wrapper) and >> >>>>> OpenJDK/Oracle >> >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11). >> >>>>>>>> * Python artifacts are deployed along with the source >> >>>>> release to the >> >>>>>>>> dist.apache.org <http://dist.apache.org/> [2]. >> >>>>>>>> >> >>>>>>>> The vote will be open for at least 72 hours. It is adopted >> >>>>> by majority >> >>>>>>>> approval, with at least 3 PMC affirmative votes. >> >>>>>>>> >> >>>>>>>> Thanks, >> >>>>>>>> JB >> >>>>>>>> >> >>>>>>>> [1] >> >>>>>>>> >> >>>>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847 >> >>>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/ >> >>>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS >> >>>>>>>> [4] >> >>>>> >> https://repository.apache.org/content/repositories/orgapachebeam-1041/ >> >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1 >> >>>>>>>> [6] https://github.com/apache/beam-site/pull/463 >> >>>>>>>> >> >>>>> >> >> >> >> -- >> >> Got feedback? go/pabloem-feedback >> <https://goto.google.com/pabloem-feedback> >> > >> > -- >> > Jean-Baptiste Onofré >> > jbono...@apache.org >> > http://blog.nanthrax.net >> > Talend - http://www.talend.com >> > -- Got feedback? go/pabloem-feedback
