Another bug: reading from PubSub with_attributes=True is broken on Python with Dataflow. https://issues.apache.org/jira/browse/BEAM-4536
JB, I'm making a PR that removes this keyword and I'd like to propose it as a cherrypick to 2.5.0. (feature should be fixed in the next release) On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath <chamik...@google.com> wrote: > FYI: looks like Python tests are failing for Windows. JIRA is > https://issues.apache.org/jira/browse/BEAM-4535. > > I don't think this is a release blocker but this should probably go in > release notes (for any user that tries to run tests on Python source > build). And we should try to incorporate a fix if we happen to cut another > release candidate for some reason. > > Thanks, > Cham > > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <pabl...@google.com> wrote: > >> Thanks everyone who has pitched in to validate the release! >> >> Boyuan Zhang and I have also run a few pipelines, and verified that they >> work properly (see release validation spreadsheet[1]). >> >> We have also found that the Game Stats pipeline is failing in Python >> Streaming Dataflow. I have filed BEAM-4534[2]. This is not a blocker, since >> Python streaming is not yet fully supported. >> >> It seems that the uploaded artifacts look good. >> >> We have noticed that the Python artifacts are still missing Python wheel >> files (compare [3] and [4]). JB, could you please add the wheel files? >> Boyuan and I can try to help you prepare them / upload them if necessary. >> Please let us know. >> >> Thanks again! >> -P. >> >> [1] >> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807 >> [2] https://issues.apache.org/jira/browse/BEAM-4534 >> [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/ >> [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/ >> >> On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold <amyrv...@google.com> >> wrote: >> >>> +1 (non-binding) >>> >>> tested some of the quickstarts >>> >>> On Sun, Jun 10, 2018 at 1:39 AM Tim <timrobertson...@gmail.com> wrote: >>> >>>> Tested by our team: >>>> - mvn inclusion >>>> - Avro, ES, Hadoop IF IO >>>> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster) >>>> - Reviewed release notes >>>> >>>> +1 >>>> >>>> Thanks also to everyone who helped get over the gradle hurdle and in >>>> particular to JB. >>>> >>>> Tim >>>> >>>> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré <j...@nanthrax.net> >>>> wrote: >>>> > >>>> > No problem Pablo. >>>> > >>>> > The vote period is a minimum, it can be extended as requested or if we >>>> > don't have the minimum of 3 binding votes. >>>> > >>>> > Regards >>>> > JB >>>> > >>>> >> On 09/06/2018 01:54, Pablo Estrada wrote: >>>> >> Hello all, >>>> >> I'd like to request an extension of the voting period until Monday >>>> >> evening (US time, so later in other geographical regions). This is >>>> >> because we were only now able to publish Dataflow Workers, and have >>>> not >>>> >> had the chance to run release validation tests on them. The extension >>>> >> will allow us to validate and vote by Monday. >>>> >> Is this acceptable to the community? >>>> >> >>>> >> Best >>>> >> -P. >>>> >> >>>> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko >>>> >> <aromanenko....@gmail.com <mailto:aromanenko....@gmail.com>> wrote: >>>> >> >>>> >> Thank you JB for your work! >>>> >> >>>> >> I tested running simple streaming (/KafkaIO/) and batch (/TextIO / >>>> >> HDFS/) pipelines with SparkRunner on YARN cluster - it works fine. >>>> >> >>>> >> WBR, >>>> >> Alexey >>>> >> >>>> >> >>>> >>> On 8 Jun 2018, at 10:00, Etienne Chauchot <echauc...@apache.org >>>> >>> <mailto:echauc...@apache.org>> wrote: >>>> >>> >>>> >>> I forgot to vote: >>>> >>> +1 (non binding). >>>> >>> What I tested: >>>> >>> - no functional or performance regression comparing to v2.4 >>>> >>> - dependencies in the poms are ok >>>> >>> >>>> >>> Etienne >>>> >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a >>>> écrit : >>>> >>>> +1 (non-binding), mainstream usage is not broken by the pom >>>> >>>> changes and runtime has no known regression compared to the >>>> 2.4.0 >>>> >>>> >>>> >>>> (side note: kudo to JB for this build tool change release, I >>>> know >>>> >>>> how it can hurt ;)) >>>> >>>> >>>> >>>> Romain Manni-Bucau >>>> >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>> >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>> >>>> <http://rmannibucau.wordpress.com/> | Github >>>> >>>> <https://github.com/rmannibucau> | LinkedIn >>>> >>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>> >>>> < >>>> https://www.packtpub.com/application-development/java-ee-8-high-performance >>>> > >>>> >>>> >>>> >>>> >>>> >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré >>>> >>>> <j...@nanthrax.net <mailto:j...@nanthrax.net>> a écrit : >>>> >>>>> Thanks for the details Etienne ! >>>> >>>>> >>>> >>>>> The good news is that the artifacts seem OK and the overall >>>> Nexmark >>>> >>>>> results are consistent with the 2.4.0 release ones. >>>> >>>>> >>>> >>>>> I'm starting a complete review using the beam-samples as well. >>>> >>>>> >>>> >>>>> Regards >>>> >>>>> JB >>>> >>>>> >>>> >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote: >>>> >>>>>> Hi, >>>> >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag >>>> >>>>>> What we can notice: >>>> >>>>>> - query 3 (exercises CoGroupByKey, state and timer) shows >>>> >>>>> different >>>> >>>>>> output with DR between batch and streaming and with the other >>>> >>>>> runners => >>>> >>>>>> I compared with v2.4 there were still these differences but with >>>> >>>>>> different output size numbers >>>> >>>>>> >>>> >>>>>> - query 6 (exercises specialized combiner) shows different output >>>> >>>>>> between the runners => the correct output is 401. strange that >>>> >>>>> in batch >>>> >>>>>> mode some runners output les Sellers. I compared with v2.4 >>>> >>>>> same output >>>> >>>>>> >>>> >>>>>> - response time of query 7 (exercices Max transform, fanout >>>> >>>>> and side >>>> >>>>>> input) is very slow on DR => I compared with v2.4 , comparable >>>> >>>>> execution >>>> >>>>>> times >>>> >>>>>> >>>> >>>>>> I'm not comparing q10 because it is a write to GCS so it is >>>> >>>>> very specific. >>>> >>>>>> >>>> >>>>>> => Basically no regression comparing to v2.4 >>>> >>>>>> >>>> >>>>>> For the record here is the output (waiting for ongoing perfkit >>>> >>>>> integration): >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> 1. DR batch >>>> >>>>>> >>>> >>>>>> Performance: >>>> >>>>>> >>>> >>>>>> >>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) >>>> Results (Baseline) >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0000 5,8 17283,1 >>>> 100000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0001 3,2 31104,2 >>>> 92000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0002 1,2 82918,7 >>>> 351 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0003 2,2 46210,7 >>>> 458 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0004 1,2 8503,4 >>>> 40 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0005 4,0 25220,7 >>>> 12 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0006 0,9 11148,3 >>>> 401 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0007 13,2 7580,9 >>>> 1 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0008 1,5 67340,1 >>>> 6000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0009 0,7 14025,2 >>>> 298 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0010 12,8 7793,0 >>>> 1 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0011 2,4 42319,1 >>>> 1919 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0012 1,6 61462,8 >>>> 1919 >>>> >>>>>> >>>> >>>>> >>>> ========================================================================================== >>>> >>>>>> >>>> >>>>>> 2. DR streaming >>>> >>>>>> >>>> >>>>>> Performance: >>>> >>>>>> >>>> >>>>>> >>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) >>>> Results (Baseline) >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0000 6,5 15285,8 >>>> 100000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0001 3,7 27397,3 >>>> 92000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0002 1,4 69108,5 >>>> 351 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0003 3,2 31181,8 >>>> 447 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0004 1,2 8361,2 >>>> 40 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0005 5,3 18903,6 >>>> 12 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0006 0,9 11111,1 >>>> 401 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0007 82,5 1212,2 >>>> 1 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0008 2,0 51072,5 >>>> 6000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0009 0,8 12903,2 >>>> 298 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0010 49,5 2021,8 >>>> 1 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0011 3,9 25667,4 >>>> 1919 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0012 2,4 41067,8 >>>> 1919 >>>> >>>>>> >>>> >>>>> >>>> ========================================================================================== >>>> >>>>>> >>>> >>>>>> 3. Flink batch >>>> >>>>>> Performance: >>>> >>>>>> >>>> >>>>>> >>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) >>>> Results (Baseline) >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0000 1,0 97656,3 >>>> 100000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0001 0,7 141643,1 >>>> 92000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0002 0,4 228310,5 >>>> 351 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0003 1,6 64020,5 >>>> 580 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0004 0,7 13831,3 >>>> 40 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0005 1,4 72939,5 >>>> 12 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0006 0,5 20491,8 >>>> 103 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0007 1,3 74239,0 >>>> 1 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0008 0,8 121506,7 >>>> 6000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0009 0,6 17953,3 >>>> 298 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0010 1,3 74682,6 >>>> 1 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0011 1,1 92936,8 >>>> 1919 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0012 0,8 123001,2 >>>> 1919 >>>> >>>>>> >>>> >>>>> >>>> ========================================================================================== >>>> >>>>>> >>>> >>>>>> 4. Flink streaming >>>> >>>>>> Performance: >>>> >>>>>> >>>> >>>>>> >>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) >>>> Results (Baseline) >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0000 5,4 18677,6 >>>> 100000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0001 2,8 35511,4 >>>> 92000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0002 1,8 54318,3 >>>> 351 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0003 2,4 41614,6 >>>> 580 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0004 1,0 10341,3 >>>> 40 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0005 3,4 29568,3 >>>> 12 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0006 0,7 13369,0 >>>> 401 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0007 2,8 36192,5 >>>> 1 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0008 1,8 54854,6 >>>> 6000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0009 0,7 13369,0 >>>> 298 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0010 3,4 29841,8 >>>> 2 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0011 5,0 19932,2 >>>> 1919 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0012 2,6 38835,0 >>>> 1919 >>>> >>>>>> >>>> >>>>> >>>> ========================================================================================== >>>> >>>>>> >>>> >>>>>> 5. Spark batch >>>> >>>>>> Performance: >>>> >>>>>> >>>> >>>>>> >>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) >>>> Results (Baseline) >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0000 1,5 65445,0 >>>> 100000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0001 1,3 79491,3 >>>> 92000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0002 0,9 112107,6 >>>> 351 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0003 2,0 48804,3 >>>> 580 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0004 1,2 8382,2 >>>> 40 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0005 2,0 50838,8 >>>> 12 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0006 1,0 9699,3 >>>> 103 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0007 2,3 43308,8 >>>> 1 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0008 2,1 46794,6 >>>> 6000 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0009 1,1 8976,7 >>>> 298 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0010 1,6 62111,8 >>>> 1 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0011 2,1 46598,3 >>>> 1919 >>>> >>>>>> >>>> >>>>>> >>>> >>>>> 0012 2,3 43687,2 >>>> 1919 >>>> >>>>>> >>>> >>>>> >>>> ========================================================================================== >>>> >>>>>> >>>> >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a >>>> écrit : >>>> >>>>>>> Thanks JB for all your work ! I believe doing the first >>>> >>>>> gradle release >>>> >>>>>>> must have been hard. >>>> >>>>>>> I'll run Nexmark on the release and keep you posted. >>>> >>>>>>> >>>> >>>>>>> Best >>>> >>>>>>> Etienne >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré >>>> >>>>> a écrit : >>>> >>>>>>>> Hi everyone, >>>> >>>>>>>> >>>> >>>>>>>> Please review and vote on the release candidate #1 for the >>>> >>>>> version >>>> >>>>>>>> 2.5.0, as follows: >>>> >>>>>>>> >>>> >>>>>>>> [ ] +1, Approve the release >>>> >>>>>>>> [ ] -1, Do not approve the release (please provide specific >>>> >>>>> comments) >>>> >>>>>>>> >>>> >>>>>>>> NB: this is the first release using Gradle, so don't be too >>>> >>>>> harsh ;) A >>>> >>>>>>>> PR about the release guide will follow thanks to this release. >>>> >>>>>>>> >>>> >>>>>>>> The complete staging area is available for your review, >>>> >>>>> which includes: >>>> >>>>>>>> * JIRA release notes [1], >>>> >>>>>>>> * the official Apache source release to be deployed to >>>> >>>>> dist.apache.org <http://dist.apache.org/> >>>> >>>>>>>> [2], which is signed with the key with fingerprint C8282E76 >>>> [3], >>>> >>>>>>>> * all artifacts to be deployed to the Maven Central >>>> >>>>> Repository [4], >>>> >>>>>>>> * source code tag "v2.5.0-RC1" [5], >>>> >>>>>>>> * website pull request listing the release and publishing >>>> >>>>> the API >>>> >>>>>>>> reference manual [6]. >>>> >>>>>>>> * Java artifacts were built with Gradle 4.7 (wrapper) and >>>> >>>>> OpenJDK/Oracle >>>> >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11). >>>> >>>>>>>> * Python artifacts are deployed along with the source >>>> >>>>> release to the >>>> >>>>>>>> dist.apache.org <http://dist.apache.org/> [2]. >>>> >>>>>>>> >>>> >>>>>>>> The vote will be open for at least 72 hours. It is adopted >>>> >>>>> by majority >>>> >>>>>>>> approval, with at least 3 PMC affirmative votes. >>>> >>>>>>>> >>>> >>>>>>>> Thanks, >>>> >>>>>>>> JB >>>> >>>>>>>> >>>> >>>>>>>> [1] >>>> >>>>>>>> >>>> >>>>> >>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847 >>>> >>>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/ >>>> >>>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS >>>> >>>>>>>> [4] >>>> >>>>> >>>> https://repository.apache.org/content/repositories/orgapachebeam-1041/ >>>> >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1 >>>> >>>>>>>> [6] https://github.com/apache/beam-site/pull/463 >>>> >>>>>>>> >>>> >>>>> >>>> >> >>>> >> -- >>>> >> Got feedback? go/pabloem-feedback >>>> <https://goto.google.com/pabloem-feedback> >>>> > >>>> > -- >>>> > Jean-Baptiste Onofré >>>> > jbono...@apache.org >>>> > http://blog.nanthrax.net >>>> > Talend - http://www.talend.com >>>> >>> -- >> Got feedback? go/pabloem-feedback >> <https://goto.google.com/pabloem-feedback> >> >
smime.p7s
Description: S/MIME Cryptographic Signature
