I think we should also leverage/invest in the automation for RC validation. We have some validation scripts, but last time I looked at them they worked only partially and had several usability issues.
On Tue, Apr 23, 2019 at 3:24 PM Ahmet Altay <[email protected]> wrote: > > > On Tue, Apr 23, 2019 at 3:21 PM Kenneth Knowles <[email protected]> wrote: > >> What can we do to make this part of day-to-day workflow instead of >> finding out during release validation? Was this just a failing test that >> was missed? >> >> Kenn >> >> On Tue, Apr 23, 2019 at 3:02 PM Andrew Pilloud <[email protected]> >> wrote: >> >>> It looks like Java Nexmark tests are on the validation sheet but we've >>> missed it the last few releases. Thanks for checking it Etienne! >>> >> > I only see nexmark perf on direct runner. Is this even a thing? Should we > also add major runners to that list? > > >> Does the current release process require everything to be tested before >>> making the release final? >>> >> > I do not think we agreed on this. Previous releases went out with partial > validations. It would be good to validate everything for the final release. > > >> >>> I fully agree with you on point 2. All of these issues were in RC1 and >>> could have been fixed for RC2. >>> >>> Andrew >>> >>> On Tue, Apr 23, 2019 at 2:58 PM Ahmet Altay <[email protected]> wrote: >>> >>>> Thank you Andrew. I will suggest two improvements to the release >>>> process: >>>> 1. We can include benchmarks in the validation sheet ("Apache Beam >>>> Release Acceptance Criteria"). They are used part of the validation process >>>> and we can ensure that we check those for each release. >>>> 2. For RC validation, we can continue to exhaustively validate each RC >>>> even after the first -1 vote. Otherwise we end up with not discovering all >>>> issues in a given RC and find them a successive RC, increasing the number >>>> of iterations required. >>>> >>>> >>>> On Tue, Apr 23, 2019 at 2:11 PM Andrew Pilloud <[email protected]> >>>> wrote: >>>> >>>>> Please consider the vote for RC4 canceled. I'll quickly follow up with >>>>> a new RC. >>>>> >>>>> Thanks for the complete testing everyone! >>>>> >>>>> Andrew >>>>> >>>>> On Tue, Apr 23, 2019 at 2:06 PM Reuven Lax <[email protected]> wrote: >>>>> >>>>>> -1 >>>>>> >>>>>> we need to cherry pick pr/8325 and pr/8385 to fix the above issue >>>>>> >>>>>> On Tue, Apr 23, 2019 at 1:48 PM Andrew Pilloud <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> I believe the breakage of Nexmark on Dataflow is >>>>>>> https://issues.apache.org/jira/browse/BEAM-7002, which went in >>>>>>> before the release was cut. It looks like this might be a release >>>>>>> blocker >>>>>>> based on the fix: https://github.com/apache/beam/pull/8325. >>>>>>> >>>>>>> The degraded performance is after the release is cut, so we should >>>>>>> be good there. >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> On Tue, Apr 23, 2019 at 8:44 AM Ismaël Mejía <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Etienne RC1 vote happened in 04/03 and there have not been any >>>>>>>> cherry >>>>>>>> picks on the spark runner afterwards so if there is a commit that >>>>>>>> degraded performance around 04/10 it is not part of the release we >>>>>>>> are >>>>>>>> voting, so please consider reverting your -1. >>>>>>>> >>>>>>>> However the issue you are reporting looks important, from a quick >>>>>>>> look >>>>>>>> I am guessing it could be related to BEAM-5775 that was merged on >>>>>>>> 12/04 however the performance regressions started happening since >>>>>>>> 09/04 so it could be unrelated. Maybe it could be due to changes in >>>>>>>> our infrastructure. Maybe the change in the workers to be tracked, >>>>>>>> but >>>>>>>> definitely not a release blocker at least for the Spark runner. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Apr 23, 2019 at 5:12 PM Etienne Chauchot < >>>>>>>> [email protected]> wrote: >>>>>>>> > >>>>>>>> > Hi guys , >>>>>>>> > I will vote -1 (binding) on this RC (although degradation is >>>>>>>> before RC4 cut date). I took a look at Nexmark graphs for the 3 major >>>>>>>> runners : >>>>>>>> > - there seem to have functional regressions on Dataflow: >>>>>>>> https://apache-beam-testing.appspot.com/explore?dashboard=5647201107705856 >>>>>>>> . 13 queries fail in batch mode starting on 04/17 >>>>>>>> > - there is a perf degradation (+200%) in spark runner starting on >>>>>>>> 04/10 for all the queries: >>>>>>>> https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712 >>>>>>>> > >>>>>>>> > Sorry Andrew for the added work >>>>>>>> > >>>>>>>> > Etienne >>>>>>>> > >>>>>>>> > Le lundi 22 avril 2019 à 12:21 -0700, Andrew Pilloud a écrit : >>>>>>>> > >>>>>>>> > I signed the wheels files and updated the build process to not >>>>>>>> require giving travis apache credentials. (You should probably change >>>>>>>> your >>>>>>>> password if you haven't already.) >>>>>>>> > >>>>>>>> > Andrew >>>>>>>> > >>>>>>>> > On Mon, Apr 22, 2019 at 12:18 PM Ahmet Altay <[email protected]> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > +1 (binding) >>>>>>>> > >>>>>>>> > Verified the python 2 wheel files with quick start examples. >>>>>>>> > >>>>>>>> > On Mon, Apr 22, 2019 at 11:26 AM Ahmet Altay <[email protected]> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > I built the wheel files. They are in the usual place along with >>>>>>>> other python artifacts. I will test them a bit and update here. Could >>>>>>>> someone else please try the wheel files as well? >>>>>>>> > >>>>>>>> > Andrew, could you sign and hash the wheel files? >>>>>>>> > >>>>>>>> > On Mon, Apr 22, 2019 at 10:11 AM Ahmet Altay <[email protected]> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > I verified >>>>>>>> > - signatures and hashes. >>>>>>>> > - python streaming quickstart guide >>>>>>>> > >>>>>>>> > I would like to verify the wheel files before voting. Please let >>>>>>>> us know when they are ready. Also, if you need help with building wheel >>>>>>>> files I can help/build. >>>>>>>> > >>>>>>>> > Ahmet >>>>>>>> > >>>>>>>> > On Mon, Apr 22, 2019 at 3:33 AM Maximilian Michels < >>>>>>>> [email protected]> wrote: >>>>>>>> > >>>>>>>> > +1 (binding) >>>>>>>> > >>>>>>>> > Found a minor bug while testing, but not a blocker: >>>>>>>> > https://jira.apache.org/jira/browse/BEAM-7128 >>>>>>>> > >>>>>>>> > Thanks, >>>>>>>> > Max >>>>>>>> > >>>>>>>> > On 20.04.19 23:02, Pablo Estrada wrote: >>>>>>>> > > +1 >>>>>>>> > > Ran SQL postcommit, and Dataflow Portability Java >>>>>>>> validatesrunner tests. >>>>>>>> > > >>>>>>>> > > -P. >>>>>>>> > > >>>>>>>> > > On Wed, Apr 17, 2019 at 1:38 AM Jean-Baptiste Onofré < >>>>>>>> [email protected] >>>>>>>> > > <mailto:[email protected]>> wrote: >>>>>>>> > > >>>>>>>> > > +1 (binding) >>>>>>>> > > >>>>>>>> > > Quickly checked with beam-samples. >>>>>>>> > > >>>>>>>> > > Regards >>>>>>>> > > JB >>>>>>>> > > >>>>>>>> > > On 16/04/2019 00:50, Andrew Pilloud wrote: >>>>>>>> > > > Hi everyone, >>>>>>>> > > > >>>>>>>> > > > Please review and vote on the release candidate #4 for >>>>>>>> the version >>>>>>>> > > > 2.12.0, as follows: >>>>>>>> > > > >>>>>>>> > > > [ ] +1, Approve the release >>>>>>>> > > > [ ] -1, Do not approve the release (please provide >>>>>>>> specific comments) >>>>>>>> > > > >>>>>>>> > > > The complete staging area is available for your review, >>>>>>>> which >>>>>>>> > > includes: >>>>>>>> > > > * JIRA release notes [1], >>>>>>>> > > > * the official Apache source release to be deployed to >>>>>>>> > > dist.apache.org <http://dist.apache.org> >>>>>>>> > > > <http://dist.apache.org> [2], which is signed with the >>>>>>>> key with >>>>>>>> > > > fingerprint 9E7CEC0661EFD610B632C610AE8FE17F9F8AE3D4 [3], >>>>>>>> > > > * all artifacts to be deployed to the Maven Central >>>>>>>> Repository [4], >>>>>>>> > > > * source code tag "v2.12.0-RC4" [5], >>>>>>>> > > > * website pull request listing the release [6], >>>>>>>> publishing the API >>>>>>>> > > > reference manual [7], and the blog post [8]. >>>>>>>> > > > * Java artifacts were built with Gradle/5.2.1 and >>>>>>>> OpenJDK/Oracle JDK >>>>>>>> > > > 1.8.0_181. >>>>>>>> > > > * Python artifacts are deployed along with the source >>>>>>>> release to the >>>>>>>> > > > dist.apache.org <http://dist.apache.org> < >>>>>>>> http://dist.apache.org> >>>>>>>> > > [2]. >>>>>>>> > > > * Validation sheet with a tab for 2.12.0 release to help >>>>>>>> with >>>>>>>> > > validation >>>>>>>> > > > [9]. >>>>>>>> > > > >>>>>>>> > > > The vote will be open for at least 72 hours. It is >>>>>>>> adopted by >>>>>>>> > > majority >>>>>>>> > > > approval, with at least 3 PMC affirmative votes. >>>>>>>> > > > >>>>>>>> > > > Thanks, >>>>>>>> > > > Andrew >>>>>>>> > > > >>>>>>>> > > > 1] >>>>>>>> > > >>>>>>>> https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12344944 >>>>>>>> > > > [2] https://dist.apache.org/repos/dist/dev/beam/2.12.0/ >>>>>>>> > > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS >>>>>>>> > > > [4] >>>>>>>> > > >>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1068/ >>>>>>>> > > > [5] https://github.com/apache/beam/tree/v2.12.0-RC4 >>>>>>>> > > > [6] https://github.com/apache/beam/pull/8215 >>>>>>>> > > > [7] https://github.com/apache/beam-site/pull/588 >>>>>>>> > > > [8] https://github.com/apache/beam/pull/8314 >>>>>>>> > > > [9] >>>>>>>> > > >>>>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1007316984 >>>>>>>> > > >>>>>>>> > > -- >>>>>>>> > > Jean-Baptiste Onofré >>>>>>>> > > [email protected] <mailto:[email protected]> >>>>>>>> > > http://blog.nanthrax.net >>>>>>>> > > Talend - http://www.talend.com >>>>>>>> > > >>>>>>>> >>>>>>>
