Hey JB, I added some instructions about how to create python wheels in this PR: https://github.com/apache/beam-site/pull/467 . Hope it would be helpful.
Boyuan On Tue, Jun 12, 2018 at 12:33 PM Jean-Baptiste Onofré <[email protected]> wrote: > Hi all, > > due to issues found by Udi and Pablo, I cancel RC1 to prepare a new > release. > > Thanks Udi and Pablo for catching up. > > I will prepare a RC2 after cherry pick/fixed the found issues (tomorrow > morning my time). > > Regards > JB > > > On 12/06/2018 21:04, Pablo Estrada wrote: > > Hello all, > > I've found another release blocker: The current findbugs dependency in > > build_rules.gradle has a GPL license. Here's the PR to depend on the > > correct findbugs: https://github.com/apache/beam/pull/5609 > > > > Best > > -P. > > > > On Tue, Jun 12, 2018 at 11:03 AM Udi Meiri <[email protected] > > <mailto:[email protected]>> wrote: > > > > Cherrypick created: https://github.com/apache/beam/pull/5607 > > Tests still running. > > > > On Tue, Jun 12, 2018 at 9:59 AM Udi Meiri <[email protected] > > <mailto:[email protected]>> wrote: > > > > -1: Would like to cherry pick a fix > > for https://issues.apache.org/jira/browse/BEAM-4536 > > > > On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré > > <[email protected] <mailto:[email protected]>> wrote: > > > > Any update about your vote and fix ? > > > > Thanks ! > > Regards > > JB > > > > On 12/06/2018 04:02, Udi Meiri wrote: > > > Another bug: reading from PubSub with_attributes=True is > > broken on > > > Python with Dataflow. > > > https://issues.apache.org/jira/browse/BEAM-4536 > > > > > > JB, I'm making a PR that removes this keyword and I'd like > > to propose it > > > as a cherrypick to 2.5.0. > > > (feature should be fixed in the next release) > > > > > > On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath > > <[email protected] <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > > > > > FYI: looks like Python tests are failing for Windows. > JIRA > > > is https://issues.apache.org/jira/browse/BEAM-4535. > > > > > > I don't think this is a release blocker but this > > should probably go > > > in release notes (for any user that tries to run tests > > on Python > > > source build). And we should try to incorporate a fix > > if we happen > > > to cut another release candidate for some reason. > > > > > > Thanks, > > > Cham > > > > > > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada > > <[email protected] <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > > > > > Thanks everyone who has pitched in to validate the > > release! > > > > > > Boyuan Zhang and I have also run a few pipelines, > > and verified > > > that they work properly (see release validation > > spreadsheet[1]). > > > > > > We have also found that the Game Stats pipeline is > > failing in > > > Python Streaming Dataflow. I have filed > > BEAM-4534[2]. This is > > > not a blocker, since Python streaming is not yet > > fully supported. > > > > > > It seems that the uploaded artifacts look good. > > > > > > We have noticed that the Python artifacts are > > still missing > > > Python wheel files (compare [3] and [4]). JB, > > could you please > > > add the wheel files? Boyuan and I can try to help > > you prepare > > > them / upload them if necessary. Please let us > know. > > > > > > Thanks again! > > > -P. > > > > > > > > [1] > https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807 > > > [2] > https://issues.apache.org/jira/browse/BEAM-4534 > > > [3] > https://dist.apache.org/repos/dist/dev/beam/2.4.0/ > > > [4] > https://dist.apache.org/repos/dist/dev/beam/2.5.0/ > > > > > > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold > > > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> > > wrote: > > > > > > +1 (non-binding) > > > > > > tested some of the quickstarts > > > > > > On Sun, Jun 10, 2018 at 1:39 AM Tim > > > <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > > > > > Tested by our team: > > > - mvn inclusion > > > - Avro, ES, Hadoop IF IO > > > - Pipelines run on Spark (Cloudera 5.12.0 > > YARN cluster) > > > - Reviewed release notes > > > > > > +1 > > > > > > Thanks also to everyone who helped get > > over the gradle > > > hurdle and in particular to JB. > > > > > > Tim > > > > > > > On 9 Jun 2018, at 05:56, Jean-Baptiste > > Onofré > > > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > > > > > No problem Pablo. > > > > > > > > The vote period is a minimum, it can be > > extended as > > > requested or if we > > > > don't have the minimum of 3 binding > votes. > > > > > > > > Regards > > > > JB > > > > > > > >> On 09/06/2018 01:54, Pablo Estrada > wrote: > > > >> Hello all, > > > >> I'd like to request an extension of the > > voting period > > > until Monday > > > >> evening (US time, so later in other > > geographical > > > regions). This is > > > >> because we were only now able to > > publish Dataflow > > > Workers, and have not > > > >> had the chance to run release > > validation tests on > > > them. The extension > > > >> will allow us to validate and vote by > > Monday. > > > >> Is this acceptable to the community? > > > >> > > > >> Best > > > >> -P. > > > >> > > > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey > > Romanenko > > > >> <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>> > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>>> wrote: > > > >> > > > >> Thank you JB for your work! > > > >> > > > >> I tested running simple streaming > > (/KafkaIO/) and > > > batch (/TextIO / > > > >> HDFS/) pipelines with SparkRunner on > > YARN cluster > > > - it works fine. > > > >> > > > >> WBR, > > > >> Alexey > > > >> > > > >> > > > >>> On 8 Jun 2018, at 10:00, Etienne > > Chauchot > > > <[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>> > > > >>> <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>>> wrote: > > > >>> > > > >>> I forgot to vote: > > > >>> +1 (non binding). > > > >>> What I tested: > > > >>> - no functional or performance > > regression > > > comparing to v2.4 > > > >>> - dependencies in the poms are ok > > > >>> > > > >>> Etienne > > > >>>> Le vendredi 08 juin 2018 à 08:27 > > +0200, Romain > > > Manni-Bucau a écrit : > > > >>>> +1 (non-binding), mainstream usage > > is not broken > > > by the pom > > > >>>> changes and runtime has no known > > regression > > > compared to the 2.4.0 > > > >>>> > > > >>>> (side note: kudo to JB for this > > build tool > > > change release, I know > > > >>>> how it can hurt ;)) > > > >>>> > > > >>>> Romain Manni-Bucau > > > >>>> @rmannibucau > > <https://twitter.com/rmannibucau> > > > | Blog > > > >>>> > > <https://rmannibucau.metawerx.net/> | Old Blog > > > >>>> > > <http://rmannibucau.wordpress.com/> | Github > > > >>>> <https://github.com/rmannibucau> | > > LinkedIn > > > >>>> > > <https://www.linkedin.com/in/rmannibucau> | Book > > > >>>> > > > > > < > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > > > >>>> > > > >>>> > > > >>>> Le jeu. 7 juin 2018 à 16:17, > > Jean-Baptiste Onofré > > > >>>> <[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>> > > > <mailto:[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>>>> a écrit : > > > >>>>> Thanks for the details Etienne ! > > > >>>>> > > > >>>>> The good news is that the > > artifacts seem OK and > > > the overall Nexmark > > > >>>>> results are consistent with the > > 2.4.0 release ones. > > > >>>>> > > > >>>>> I'm starting a complete review > > using the > > > beam-samples as well. > > > >>>>> > > > >>>>> Regards > > > >>>>> JB > > > >>>>> > > > >>>>>> On 07/06/2018 16:14, Etienne > > Chauchot wrote: > > > >>>>>> Hi, > > > >>>>>> I've just run the nexmark queries > > on v2.5.0-RC1 tag > > > >>>>>> What we can notice: > > > >>>>>> - query 3 (exercises CoGroupByKey, > > state and > > > timer) shows > > > >>>>> different > > > >>>>>> output with DR between batch and > > streaming and > > > with the other > > > >>>>> runners => > > > >>>>>> I compared with v2.4 there were > > still these > > > differences but with > > > >>>>>> different output size numbers > > > >>>>>> > > > >>>>>> - query 6 (exercises specialized > > combiner) shows > > > different output > > > >>>>>> between the runners => the correct > > output is 401. > > > strange that > > > >>>>> in batch > > > >>>>>> mode some runners output les > > Sellers. I compared > > > with v2.4 > > > >>>>> same output > > > >>>>>> > > > >>>>>> - response time of query 7 > > (exercices Max > > > transform, fanout > > > >>>>> and side > > > >>>>>> input) is very slow on DR => I > > compared with v2.4 > > > , comparable > > > >>>>> execution > > > >>>>>> times > > > >>>>>> > > > >>>>>> I'm not comparing q10 because it is > > a write to > > > GCS so it is > > > >>>>> very specific. > > > >>>>>> > > > >>>>>> => Basically no regression > > comparing to v2.4 > > > >>>>>> > > > >>>>>> For the record here is the output > > (waiting for > > > ongoing perfkit > > > >>>>> integration): > > > >>>>>> > > > >>>>>> > > > >>>>>> 1. DR batch > > > >>>>>> > > > >>>>>> Performance: > > > >>>>>> > > > >>>>>> > > > >>>>> Conf Runtime(sec) (Baseline) > > Events(/sec) > > > (Baseline) Results (Baseline) > > > >>>>>> > > > >>>>>> > > > >>>>> 0000 5,8 > > 17283,1 > > > 100000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0001 3,2 > > 31104,2 > > > 92000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0002 1,2 > > 82918,7 > > > 351 > > > >>>>>> > > > >>>>>> > > > >>>>> 0003 2,2 > > 46210,7 > > > 458 > > > >>>>>> > > > >>>>>> > > > >>>>> 0004 1,2 > > 8503,4 > > > 40 > > > >>>>>> > > > >>>>>> > > > >>>>> 0005 4,0 > > 25220,7 > > > 12 > > > >>>>>> > > > >>>>>> > > > >>>>> 0006 0,9 > > 11148,3 > > > 401 > > > >>>>>> > > > >>>>>> > > > >>>>> 0007 13,2 > > 7580,9 > > > 1 > > > >>>>>> > > > >>>>>> > > > >>>>> 0008 1,5 > > 67340,1 > > > 6000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0009 0,7 > > 14025,2 > > > 298 > > > >>>>>> > > > >>>>>> > > > >>>>> 0010 12,8 > > 7793,0 > > > 1 > > > >>>>>> > > > >>>>>> > > > >>>>> 0011 2,4 > > 42319,1 > > > 1919 > > > >>>>>> > > > >>>>>> > > > >>>>> 0012 1,6 > > 61462,8 > > > 1919 > > > >>>>>> > > > >>>>> > > > > > > > ========================================================================================== > > > >>>>>> > > > >>>>>> 2. DR streaming > > > >>>>>> > > > >>>>>> Performance: > > > >>>>>> > > > >>>>>> > > > >>>>> Conf Runtime(sec) (Baseline) > > Events(/sec) > > > (Baseline) Results (Baseline) > > > >>>>>> > > > >>>>>> > > > >>>>> 0000 6,5 > > 15285,8 > > > 100000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0001 3,7 > > 27397,3 > > > 92000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0002 1,4 > > 69108,5 > > > 351 > > > >>>>>> > > > >>>>>> > > > >>>>> 0003 3,2 > > 31181,8 > > > 447 > > > >>>>>> > > > >>>>>> > > > >>>>> 0004 1,2 > > 8361,2 > > > 40 > > > >>>>>> > > > >>>>>> > > > >>>>> 0005 5,3 > > 18903,6 > > > 12 > > > >>>>>> > > > >>>>>> > > > >>>>> 0006 0,9 > > 11111,1 > > > 401 > > > >>>>>> > > > >>>>>> > > > >>>>> 0007 82,5 > > 1212,2 > > > 1 > > > >>>>>> > > > >>>>>> > > > >>>>> 0008 2,0 > > 51072,5 > > > 6000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0009 0,8 > > 12903,2 > > > 298 > > > >>>>>> > > > >>>>>> > > > >>>>> 0010 49,5 > > 2021,8 > > > 1 > > > >>>>>> > > > >>>>>> > > > >>>>> 0011 3,9 > > 25667,4 > > > 1919 > > > >>>>>> > > > >>>>>> > > > >>>>> 0012 2,4 > > 41067,8 > > > 1919 > > > >>>>>> > > > >>>>> > > > > > > > ========================================================================================== > > > >>>>>> > > > >>>>>> 3. Flink batch > > > >>>>>> Performance: > > > >>>>>> > > > >>>>>> > > > >>>>> Conf Runtime(sec) (Baseline) > > Events(/sec) > > > (Baseline) Results (Baseline) > > > >>>>>> > > > >>>>>> > > > >>>>> 0000 1,0 > > 97656,3 > > > 100000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0001 0,7 > > 141643,1 > > > 92000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0002 0,4 > > 228310,5 > > > 351 > > > >>>>>> > > > >>>>>> > > > >>>>> 0003 1,6 > > 64020,5 > > > 580 > > > >>>>>> > > > >>>>>> > > > >>>>> 0004 0,7 > > 13831,3 > > > 40 > > > >>>>>> > > > >>>>>> > > > >>>>> 0005 1,4 > > 72939,5 > > > 12 > > > >>>>>> > > > >>>>>> > > > >>>>> 0006 0,5 > > 20491,8 > > > 103 > > > >>>>>> > > > >>>>>> > > > >>>>> 0007 1,3 > > 74239,0 > > > 1 > > > >>>>>> > > > >>>>>> > > > >>>>> 0008 0,8 > > 121506,7 > > > 6000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0009 0,6 > > 17953,3 > > > 298 > > > >>>>>> > > > >>>>>> > > > >>>>> 0010 1,3 > > 74682,6 > > > 1 > > > >>>>>> > > > >>>>>> > > > >>>>> 0011 1,1 > > 92936,8 > > > 1919 > > > >>>>>> > > > >>>>>> > > > >>>>> 0012 0,8 > > 123001,2 > > > 1919 > > > >>>>>> > > > >>>>> > > > > > > > ========================================================================================== > > > >>>>>> > > > >>>>>> 4. Flink streaming > > > >>>>>> Performance: > > > >>>>>> > > > >>>>>> > > > >>>>> Conf Runtime(sec) (Baseline) > > Events(/sec) > > > (Baseline) Results (Baseline) > > > >>>>>> > > > >>>>>> > > > >>>>> 0000 5,4 > > 18677,6 > > > 100000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0001 2,8 > > 35511,4 > > > 92000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0002 1,8 > > 54318,3 > > > 351 > > > >>>>>> > > > >>>>>> > > > >>>>> 0003 2,4 > > 41614,6 > > > 580 > > > >>>>>> > > > >>>>>> > > > >>>>> 0004 1,0 > > 10341,3 > > > 40 > > > >>>>>> > > > >>>>>> > > > >>>>> 0005 3,4 > > 29568,3 > > > 12 > > > >>>>>> > > > >>>>>> > > > >>>>> 0006 0,7 > > 13369,0 > > > 401 > > > >>>>>> > > > >>>>>> > > > >>>>> 0007 2,8 > > 36192,5 > > > 1 > > > >>>>>> > > > >>>>>> > > > >>>>> 0008 1,8 > > 54854,6 > > > 6000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0009 0,7 > > 13369,0 > > > 298 > > > >>>>>> > > > >>>>>> > > > >>>>> 0010 3,4 > > 29841,8 > > > 2 > > > >>>>>> > > > >>>>>> > > > >>>>> 0011 5,0 > > 19932,2 > > > 1919 > > > >>>>>> > > > >>>>>> > > > >>>>> 0012 2,6 > > 38835,0 > > > 1919 > > > >>>>>> > > > >>>>> > > > > > > > ========================================================================================== > > > >>>>>> > > > >>>>>> 5. Spark batch > > > >>>>>> Performance: > > > >>>>>> > > > >>>>>> > > > >>>>> Conf Runtime(sec) (Baseline) > > Events(/sec) > > > (Baseline) Results (Baseline) > > > >>>>>> > > > >>>>>> > > > >>>>> 0000 1,5 > > 65445,0 > > > 100000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0001 1,3 > > 79491,3 > > > 92000 > > > >>>>>> > > > >>>>>> > > > >>>>> 0002 0,9 > > 112107,6 > > > 351 > > > >>>>>> > > > >>>>>> > > > >>>>> 0003 2,0 > > 48804,3 > > > 580 > > > >>
