Hello all, I've found another release blocker: The current findbugs dependency in build_rules.gradle has a GPL license. Here's the PR to depend on the correct findbugs: https://github.com/apache/beam/pull/5609
Best -P. On Tue, Jun 12, 2018 at 11:03 AM Udi Meiri <[email protected]> wrote: > Cherrypick created: https://github.com/apache/beam/pull/5607 > Tests still running. > > On Tue, Jun 12, 2018 at 9:59 AM Udi Meiri <[email protected]> wrote: > >> -1: Would like to cherry pick a fix for >> https://issues.apache.org/jira/browse/BEAM-4536 >> >> On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré <[email protected]> >> wrote: >> >>> Any update about your vote and fix ? >>> >>> Thanks ! >>> Regards >>> JB >>> >>> On 12/06/2018 04:02, Udi Meiri wrote: >>> > Another bug: reading from PubSub with_attributes=True is broken on >>> > Python with Dataflow. >>> > https://issues.apache.org/jira/browse/BEAM-4536 >>> > >>> > JB, I'm making a PR that removes this keyword and I'd like to propose >>> it >>> > as a cherrypick to 2.5.0. >>> > (feature should be fixed in the next release) >>> > >>> > On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath < >>> [email protected] >>> > <mailto:[email protected]>> wrote: >>> > >>> > FYI: looks like Python tests are failing for Windows. JIRA >>> > is https://issues.apache.org/jira/browse/BEAM-4535. >>> > >>> > I don't think this is a release blocker but this should probably go >>> > in release notes (for any user that tries to run tests on Python >>> > source build). And we should try to incorporate a fix if we happen >>> > to cut another release candidate for some reason. >>> > >>> > Thanks, >>> > Cham >>> > >>> > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <[email protected] >>> > <mailto:[email protected]>> wrote: >>> > >>> > Thanks everyone who has pitched in to validate the release! >>> > >>> > Boyuan Zhang and I have also run a few pipelines, and verified >>> > that they work properly (see release validation >>> spreadsheet[1]). >>> > >>> > We have also found that the Game Stats pipeline is failing in >>> > Python Streaming Dataflow. I have filed BEAM-4534[2]. This is >>> > not a blocker, since Python streaming is not yet fully >>> supported. >>> > >>> > It seems that the uploaded artifacts look good. >>> > >>> > We have noticed that the Python artifacts are still missing >>> > Python wheel files (compare [3] and [4]). JB, could you please >>> > add the wheel files? Boyuan and I can try to help you prepare >>> > them / upload them if necessary. Please let us know. >>> > >>> > Thanks again! >>> > -P. >>> > >>> > [1] >>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807 >>> > [2] https://issues.apache.org/jira/browse/BEAM-4534 >>> > [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/ >>> > [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/ >>> > >>> > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold >>> > <[email protected] <mailto:[email protected]>> wrote: >>> > >>> > +1 (non-binding) >>> > >>> > tested some of the quickstarts >>> > >>> > On Sun, Jun 10, 2018 at 1:39 AM Tim >>> > <[email protected] >>> > <mailto:[email protected]>> wrote: >>> > >>> > Tested by our team: >>> > - mvn inclusion >>> > - Avro, ES, Hadoop IF IO >>> > - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster) >>> > - Reviewed release notes >>> > >>> > +1 >>> > >>> > Thanks also to everyone who helped get over the gradle >>> > hurdle and in particular to JB. >>> > >>> > Tim >>> > >>> > > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré >>> > <[email protected] <mailto:[email protected]>> wrote: >>> > > >>> > > No problem Pablo. >>> > > >>> > > The vote period is a minimum, it can be extended as >>> > requested or if we >>> > > don't have the minimum of 3 binding votes. >>> > > >>> > > Regards >>> > > JB >>> > > >>> > >> On 09/06/2018 01:54, Pablo Estrada wrote: >>> > >> Hello all, >>> > >> I'd like to request an extension of the voting >>> period >>> > until Monday >>> > >> evening (US time, so later in other geographical >>> > regions). This is >>> > >> because we were only now able to publish Dataflow >>> > Workers, and have not >>> > >> had the chance to run release validation tests on >>> > them. The extension >>> > >> will allow us to validate and vote by Monday. >>> > >> Is this acceptable to the community? >>> > >> >>> > >> Best >>> > >> -P. >>> > >> >>> > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko >>> > >> <[email protected] >>> > <mailto:[email protected]> >>> > <mailto:[email protected] >>> > <mailto:[email protected]>>> wrote: >>> > >> >>> > >> Thank you JB for your work! >>> > >> >>> > >> I tested running simple streaming (/KafkaIO/) and >>> > batch (/TextIO / >>> > >> HDFS/) pipelines with SparkRunner on YARN cluster >>> > - it works fine. >>> > >> >>> > >> WBR, >>> > >> Alexey >>> > >> >>> > >> >>> > >>> On 8 Jun 2018, at 10:00, Etienne Chauchot >>> > <[email protected] <mailto:[email protected]> >>> > >>> <mailto:[email protected] >>> > <mailto:[email protected]>>> wrote: >>> > >>> >>> > >>> I forgot to vote: >>> > >>> +1 (non binding). >>> > >>> What I tested: >>> > >>> - no functional or performance regression >>> > comparing to v2.4 >>> > >>> - dependencies in the poms are ok >>> > >>> >>> > >>> Etienne >>> > >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain >>> > Manni-Bucau a écrit : >>> > >>>> +1 (non-binding), mainstream usage is not >>> broken >>> > by the pom >>> > >>>> changes and runtime has no known regression >>> > compared to the 2.4.0 >>> > >>>> >>> > >>>> (side note: kudo to JB for this build tool >>> > change release, I know >>> > >>>> how it can hurt ;)) >>> > >>>> >>> > >>>> Romain Manni-Bucau >>> > >>>> @rmannibucau <https://twitter.com/rmannibucau> >>> > | Blog >>> > >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>> > >>>> <http://rmannibucau.wordpress.com/> | Github >>> > >>>> <https://github.com/rmannibucau> | LinkedIn >>> > >>>> <https://www.linkedin.com/in/rmannibucau> | >>> Book >>> > >>>> >>> > < >>> https://www.packtpub.com/application-development/java-ee-8-high-performance >>> > >>> > >>>> >>> > >>>> >>> > >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste >>> Onofré >>> > >>>> <[email protected] <mailto:[email protected]> >>> > <mailto:[email protected] <mailto:[email protected]>>> a >>> écrit : >>> > >>>>> Thanks for the details Etienne ! >>> > >>>>> >>> > >>>>> The good news is that the artifacts seem OK >>> and >>> > the overall Nexmark >>> > >>>>> results are consistent with the 2.4.0 release >>> ones. >>> > >>>>> >>> > >>>>> I'm starting a complete review using the >>> > beam-samples as well. >>> > >>>>> >>> > >>>>> Regards >>> > >>>>> JB >>> > >>>>> >>> > >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote: >>> > >>>>>> Hi, >>> > >>>>>> I've just run the nexmark queries on v2.5.0-RC1 >>> tag >>> > >>>>>> What we can notice: >>> > >>>>>> - query 3 (exercises CoGroupByKey, state and >>> > timer) shows >>> > >>>>> different >>> > >>>>>> output with DR between batch and streaming and >>> > with the other >>> > >>>>> runners => >>> > >>>>>> I compared with v2.4 there were still these >>> > differences but with >>> > >>>>>> different output size numbers >>> > >>>>>> >>> > >>>>>> - query 6 (exercises specialized combiner) shows >>> > different output >>> > >>>>>> between the runners => the correct output is >>> 401. >>> > strange that >>> > >>>>> in batch >>> > >>>>>> mode some runners output les Sellers. I compared >>> > with v2.4 >>> > >>>>> same output >>> > >>>>>> >>> > >>>>>> - response time of query 7 (exercices Max >>> > transform, fanout >>> > >>>>> and side >>> > >>>>>> input) is very slow on DR => I compared with >>> v2.4 >>> > , comparable >>> > >>>>> execution >>> > >>>>>> times >>> > >>>>>> >>> > >>>>>> I'm not comparing q10 because it is a write to >>> > GCS so it is >>> > >>>>> very specific. >>> > >>>>>> >>> > >>>>>> => Basically no regression comparing to v2.4 >>> > >>>>>> >>> > >>>>>> For the record here is the output (waiting for >>> > ongoing perfkit >>> > >>>>> integration): >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> 1. DR batch >>> > >>>>>> >>> > >>>>>> Performance: >>> > >>>>>> >>> > >>>>>> >>> > >>>>> Conf Runtime(sec) (Baseline) >>> Events(/sec) >>> > (Baseline) Results (Baseline) >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0000 5,8 >>> 17283,1 >>> > 100000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0001 3,2 >>> 31104,2 >>> > 92000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0002 1,2 >>> 82918,7 >>> > 351 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0003 2,2 >>> 46210,7 >>> > 458 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0004 1,2 >>> 8503,4 >>> > 40 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0005 4,0 >>> 25220,7 >>> > 12 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0006 0,9 >>> 11148,3 >>> > 401 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0007 13,2 >>> 7580,9 >>> > 1 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0008 1,5 >>> 67340,1 >>> > 6000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0009 0,7 >>> 14025,2 >>> > 298 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0010 12,8 >>> 7793,0 >>> > 1 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0011 2,4 >>> 42319,1 >>> > 1919 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0012 1,6 >>> 61462,8 >>> > 1919 >>> > >>>>>> >>> > >>>>> >>> > >>> >>> ========================================================================================== >>> > >>>>>> >>> > >>>>>> 2. DR streaming >>> > >>>>>> >>> > >>>>>> Performance: >>> > >>>>>> >>> > >>>>>> >>> > >>>>> Conf Runtime(sec) (Baseline) >>> Events(/sec) >>> > (Baseline) Results (Baseline) >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0000 6,5 >>> 15285,8 >>> > 100000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0001 3,7 >>> 27397,3 >>> > 92000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0002 1,4 >>> 69108,5 >>> > 351 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0003 3,2 >>> 31181,8 >>> > 447 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0004 1,2 >>> 8361,2 >>> > 40 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0005 5,3 >>> 18903,6 >>> > 12 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0006 0,9 >>> 11111,1 >>> > 401 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0007 82,5 >>> 1212,2 >>> > 1 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0008 2,0 >>> 51072,5 >>> > 6000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0009 0,8 >>> 12903,2 >>> > 298 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0010 49,5 >>> 2021,8 >>> > 1 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0011 3,9 >>> 25667,4 >>> > 1919 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0012 2,4 >>> 41067,8 >>> > 1919 >>> > >>>>>> >>> > >>>>> >>> > >>> >>> ========================================================================================== >>> > >>>>>> >>> > >>>>>> 3. Flink batch >>> > >>>>>> Performance: >>> > >>>>>> >>> > >>>>>> >>> > >>>>> Conf Runtime(sec) (Baseline) >>> Events(/sec) >>> > (Baseline) Results (Baseline) >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0000 1,0 >>> 97656,3 >>> > 100000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0001 0,7 >>> 141643,1 >>> > 92000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0002 0,4 >>> 228310,5 >>> > 351 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0003 1,6 >>> 64020,5 >>> > 580 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0004 0,7 >>> 13831,3 >>> > 40 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0005 1,4 >>> 72939,5 >>> > 12 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0006 0,5 >>> 20491,8 >>> > 103 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0007 1,3 >>> 74239,0 >>> > 1 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0008 0,8 >>> 121506,7 >>> > 6000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0009 0,6 >>> 17953,3 >>> > 298 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0010 1,3 >>> 74682,6 >>> > 1 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0011 1,1 >>> 92936,8 >>> > 1919 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0012 0,8 >>> 123001,2 >>> > 1919 >>> > >>>>>> >>> > >>>>> >>> > >>> >>> ========================================================================================== >>> > >>>>>> >>> > >>>>>> 4. Flink streaming >>> > >>>>>> Performance: >>> > >>>>>> >>> > >>>>>> >>> > >>>>> Conf Runtime(sec) (Baseline) >>> Events(/sec) >>> > (Baseline) Results (Baseline) >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0000 5,4 >>> 18677,6 >>> > 100000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0001 2,8 >>> 35511,4 >>> > 92000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0002 1,8 >>> 54318,3 >>> > 351 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0003 2,4 >>> 41614,6 >>> > 580 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0004 1,0 >>> 10341,3 >>> > 40 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0005 3,4 >>> 29568,3 >>> > 12 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0006 0,7 >>> 13369,0 >>> > 401 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0007 2,8 >>> 36192,5 >>> > 1 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0008 1,8 >>> 54854,6 >>> > 6000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0009 0,7 >>> 13369,0 >>> > 298 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0010 3,4 >>> 29841,8 >>> > 2 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0011 5,0 >>> 19932,2 >>> > 1919 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0012 2,6 >>> 38835,0 >>> > 1919 >>> > >>>>>> >>> > >>>>> >>> > >>> >>> ========================================================================================== >>> > >>>>>> >>> > >>>>>> 5. Spark batch >>> > >>>>>> Performance: >>> > >>>>>> >>> > >>>>>> >>> > >>>>> Conf Runtime(sec) (Baseline) >>> Events(/sec) >>> > (Baseline) Results (Baseline) >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0000 1,5 >>> 65445,0 >>> > 100000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0001 1,3 >>> 79491,3 >>> > 92000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0002 0,9 >>> 112107,6 >>> > 351 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0003 2,0 >>> 48804,3 >>> > 580 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0004 1,2 >>> 8382,2 >>> > 40 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0005 2,0 >>> 50838,8 >>> > 12 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0006 1,0 >>> 9699,3 >>> > 103 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0007 2,3 >>> 43308,8 >>> > 1 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0008 2,1 >>> 46794,6 >>> > 6000 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0009 1,1 >>> 8976,7 >>> > 298 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0010 1,6 >>> 62111,8 >>> > 1 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0011 2,1 >>> 46598,3 >>> > 1919 >>> > >>>>>> >>> > >>>>>> >>> > >>>>> 0012 2,3 >>> 43687,2 >>> > 1919 >>> > >>>>>> >>> > >>>>> >>> > >>> >>> ========================================================================================== >>> > >>>>>> >>> > >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne >>> > Chauchot a écrit : >>> > >>>>>>> Thanks JB for all your work ! I believe doing >>> > the first >>> > >>>>> gradle release >>> > >>>>>>> must have been hard. >>> > >>>>>>> I'll run Nexmark on the release and keep you >>> posted. >>> > >>>>>>> >>> > >>>>>>> Best >>> > >>>>>>> Etienne >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, >>> > Jean-Baptiste Onofré >>> > >>>>> a écrit : >>> > >>>>>>>> Hi everyone, >>> > >>>>>>>> >>> > >>>>>>>> Please review and vote on the release >>> candidate >>> > #1 for the >>> > >>>>> version >>> > >>>>>>>> 2.5.0, as follows: >>> > >>>>>>>> >>> > >>>>>>>> [ ] +1, Approve the release >>> > >>>>>>>> [ ] -1, Do not approve the release (please >>> > provide specific >>> > >>>>> comments) >>> > >>>>>>>> >>> > >>>>>>>> NB: this is the first release using Gradle, so >>> > don't be too >>> > >>>>> harsh ;) A >>> > >>>>>>>> PR about the release guide will follow thanks >>> > to this release. >>> > >>>>>>>> >>> > >>>>>>>> The complete staging area is available for >>> your >>> > review, >>> > >>>>> which includes: >>> > >>>>>>>> * JIRA release notes [1], >>> > >>>>>>>> * the official Apache source release to be >>> > deployed to >>> > >>>>> dist.apache.org <http://dist.apache.org> >>> > <http://dist.apache.org/> >>> > >>>>>>>> [2], which is signed with the key with >>> > fingerprint C8282E76 [3], >>> > >>>>>>>> * all artifacts to be deployed to the Maven >>> Central >>> > >>>>> Repository [4], >>> > >>>>>>>> * source code tag "v2.5.0-RC1" [5], >>> > >>>>>>>> * website pull request listing the release and >>> > publishing >>> > >>>>> the API >>> > >>>>>>>> reference manual [6]. >>> > >>>>>>>> * Java artifacts were built with Gradle 4.7 >>> > (wrapper) and >>> > >>>>> OpenJDK/Oracle >>> > >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11). >>> > >>>>>>>> * Python artifacts are deployed along with the >>> > source >>> > >>>>> release to the >>> > >>>>>>>> dist.apache.org <http://dist.apache.org> >>> > <http://dist.apache.org/> [2]. >>> > >>>>>>>> >>> > >>>>>>>> The vote will be open for at least 72 hours. >>> It >>> > is adopted >>> > >>>>> by majority >>> > >>>>>>>> approval, with at least 3 PMC affirmative >>> votes. >>> > >>>>>>>> >>> > >>>>>>>> Thanks, >>> > >>>>>>>> JB >>> > >>>>>>>> >>> > >>>>>>>> [1] >>> > >>>>>>>> >>> > >>>>> >>> > >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847 >>> > >>>>>>>> [2] >>> > https://dist.apache.org/repos/dist/dev/beam/2.5.0/ >>> > >>>>>>>> [3] >>> > https://dist.apache.org/repos/dist/release/beam/KEYS >>> > >>>>>>>> [4] >>> > >>>>> >>> > >>> https://repository.apache.org/content/repositories/orgapachebeam-1041/ >>> > >>>>>>>> [5] >>> https://github.com/apache/beam/tree/v2.5.0-RC1 >>> > >>>>>>>> [6] >>> https://github.com/apache/beam-site/pull/463 >>> > >>>>>>>> >>> > >>>>> >>> > >> >>> > >> -- >>> > >> Got feedback? go/pabloem-feedback >>> <https://goto.google.com/pabloem-feedback> >>> > <https://goto.google.com/pabloem-feedback> >>> > > >>> > > -- >>> > > Jean-Baptiste Onofré >>> > > [email protected] <mailto:[email protected]> >>> > > http://blog.nanthrax.net >>> > > Talend - http://www.talend.com >>> > >>> > -- >>> > Got feedback? go/pabloem-feedback >>> <https://goto.google.com/pabloem-feedback> >>> > <https://goto.google.com/pabloem-feedback> >>> > >>> >>> -- >>> Jean-Baptiste Onofré >>> [email protected] >>> http://blog.nanthrax.net >>> Talend - http://www.talend.com >>> >> -- Got feedback? go/pabloem-feedback
