When trying to upgrade our fork from 2.16 to 2.18, we see a significant performance degradation. This applies to Flink portable streaming with the Python SDK. I don't know what the cause is yet.
If anyone else has done validation with a similar setup and this RC, it would be good to know the results. Thomas On Wed, Jan 22, 2020, 7:42 AM Valentyn Tymofieiev <valen...@google.com> wrote: > +1. > In addition to checks performed earlier, re-ran streaming quickstarts and > batch mobile gaming examples on Dataflow runner after containers were > updated. > > On Wed, Jan 22, 2020 at 7:33 AM Ismaël Mejía <ieme...@gmail.com> wrote: > >> +1 (binding) >> >> - Validated signatures >> - Run Python wordcount on Direct runner (from wheels) >> - Run Python wordcount on Flink runner with job-server image (via wheels) >> - Run Python wordcount on Spark runner with job-server from source (via >> wheels) >> - Validate no regressions on Nexmark for Spark classic runner >> - Validate provided artifacts in two external projects beam-samples + one >> internal company project >> >> Thanks Kyle, I run it as you said and everything worked but there was a >> weird exception on removal of a directory, but everything worked. >> +1 to update release validation guide/script looks worth for this case. >> >> >> On Wed, Jan 22, 2020 at 1:59 AM Kyle Weaver <kcwea...@google.com> wrote: >> >>> > Also, does anyone know how can I (we) validate the new docker image >>> for Flink's job server included in this release? >>> >>> To start the job server: >>> >>> docker run --net=host apachebeam/flink1.9_job_server:2.18.0_rc1 >>> >>> Then you can run any Beam Java/Python/Go job with job endpoint >>> localhost:8099 to validate. >>> >>> I can update the release validation script/guide with more instructions. >>> >>> On Tue, Jan 21, 2020 at 1:48 PM Robert Bradshaw <rober...@google.com> >>> wrote: >>> >>>> On Tue, Jan 21, 2020 at 12:04 PM Ahmet Altay <al...@google.com> wrote: >>>> > >>>> > This change (https://github.com/apache/beam/pull/10625) was merged >>>> after the RC1 email was out. IMO, we do not need to block RC1 vote for >>>> this. If there will be an RC2 the change will be included. >>>> >>>> Agreed, we do not need to block RC1 due to this PR that didn't make >>>> it. Just wanted to confirm that it wasn't an oversight. >>>> >>>> The signatures and wheels look good to me. +1 (binding). >>>> >>>> > I recall we had a similar thread before. Please, include the release >>>> managers in the PRs that are/will be merged into the release branch and tag >>>> JIRA issues with the all relevant releases that should be blocked on it. >>>> > >>>> > Ahmet >>>> > >>>> > On Tue, Jan 21, 2020 at 11:36 AM Udi Meiri <eh...@google.com> wrote: >>>> >> >>>> >> I was not aware of https://issues.apache.org/jira/browse/BEAM-9123 >>>> or the PR on the release branch. >>>> >> >>>> >> On Tue, Jan 21, 2020 at 11:16 AM Robert Bradshaw < >>>> rober...@google.com> wrote: >>>> >>> >>>> >>> The source tarball seems to be missing the commit at >>>> >>> >>>> https://github.com/apache/beam/commit/a61dfbf4570e3adb30e15315c116751faeda897e >>>> >>> >>>> >>> On Tue, Jan 21, 2020 at 9:49 AM Ahmet Altay <al...@google.com> >>>> wrote: >>>> >>> > >>>> >>> > All, could you help with validations and voting? >>>> >>> > >>>> >>> > On Wed, Jan 15, 2020 at 6:14 PM Ahmet Altay <al...@google.com> >>>> wrote: >>>> >>> >> >>>> >>> >> +1, validated the same things, they still work. Thank you. >>>> >>> >> >>>> >>> >> On Wed, Jan 15, 2020 at 5:01 PM Udi Meiri <eh...@google.com> >>>> wrote: >>>> >>> >>> >>>> >>> >>> Dataflow containers have been updated. Test away. >>>> >>> >>> >>>> >>> >>> On Tue, Jan 14, 2020 at 6:37 PM Udi Meiri <eh...@google.com> >>>> wrote: >>>> >>> >>>> >>>> >>> >>>> Here my second take: >>>> >>> >>>> >>>> >>> >>>> Hi everyone, >>>> >>> >>>> Please review and vote on the release candidate #1 for the >>>> version 2.18.0, as follows: >>>> >>> >>>> [ ] +1, Approve the release >>>> >>> >>>> [ ] -1, Do not approve the release (please provide specific >>>> comments) >>>> >>> >>>> >>>> >>> >>>> The complete staging area is available for your review, which >>>> includes: >>>> >>> >>>> * JIRA release notes [1], >>>> >>> >>>> * the official Apache source release to be deployed to >>>> dist.apache.org [2], which is signed with the key with fingerprint >>>> 8961 F3EF 8E79 6688 4067 87CF 587B 049C 36DA AFE6 [3], >>>> >>> >>>> * all artifacts to be deployed to the Maven Central Repository >>>> [4], >>>> >>> >>>> * source code tag "v2.18.0-RC1" [5], >>>> >>> >>>> * website pull request listing the release [6], publishing the >>>> API reference manual [7], and the blog post [8]. >>>> >>> >>>> * Java artifacts were built with Maven N/A and OpenJDK >>>> 1.8.0_181-google-v7. >>>> >>> >>>> * Python artifacts are deployed along with the source release >>>> to the dist.apache.org [2]. >>>> >>> >>>> * Validation sheet with a tab for 2.18.0 release to help with >>>> validation [9]. >>>> >>> >>>> * Docker images published to Docker Hub [10]. >>>> >>> >>>> >>>> >>> >>>> The vote will be open for at least 72 hours. It is adopted by >>>> majority approval, with at least 3 PMC affirmative votes. >>>> >>> >>>> NOTE: The vote will start once new Dataflow containers are >>>> built. >>>> >>> >>>> >>>> >>> >>>> Thanks, >>>> >>> >>>> Release Manager >>>> >>> >>>> >>>> >>> >>>> [1] >>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12346383&projectId=12319527 >>>> >>> >>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.18.0/ >>>> >>> >>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS >>>> >>> >>>> [4] >>>> https://repository.apache.org/content/repositories/orgapachebeam-1090/ >>>> >>> >>>> [5] https://github.com/apache/beam/tree/v2.18.0-RC1 >>>> >>> >>>> [6] https://github.com/apache/beam/pull/10574 >>>> >>> >>>> [7] https://github.com/apache/beam-site/pull/595 >>>> >>> >>>> [8] https://github.com/apache/beam/pull/10575 >>>> >>> >>>> [9] >>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1178617819 >>>> >>> >>>> [10] https://hub.docker.com/u/apachebeam >>>> >>> >>>> >>>> >>> >>>> >>>> >>> >>>> On Tue, Jan 14, 2020 at 6:34 PM Udi Meiri <eh...@google.com> >>>> wrote: >>>> >>> >>>>> >>>> >>> >>>>> Please don't do any Dataflow-based verifications yet, because >>>> we'll have to redo them once new Dataflow containers are built. >>>> >>> >>>>> >>>> >>> >>>>> On Tue, Jan 14, 2020 at 6:27 PM Ahmet Altay <al...@google.com> >>>> wrote: >>>> >>> >>>>>> >>>> >>> >>>>>> I verified python 2 quickstarts with batch and streaming >>>> pipelines, wheel files, and reviewed changes to the blog/website. >>>> >>> >>>>>> >>>> >>> >>>>>> Udi, could you send an updated version of the voting text >>>> with TODOs, template pieces removed? We can discuss changes to the template >>>> separately. My vote is +1 pending an updated vote text. >>>> >>> >>>>>> >>>> >>> >>>>>> On Tue, Jan 14, 2020 at 4:47 PM Udi Meiri <eh...@google.com> >>>> wrote: >>>> >>> >>>>>>> >>>> >>> >>>>>>> Sorry about the messiness. >>>> >>> >>>>>>> The links at the bottom should be correct though. >>>> >>> >>>>>>> >>>> >>> >>>>>>> I intentionally did not replace MAVEN_VERSION because I >>>> didn't know how to get it (I didn't execute mvn for the release). >>>> >>> >>>>>>> As for JDK_VERSION, do we still need that? (If so, what >>>> about Python versions, such as the ones used for testing?) >>>> >>> >>>>>>> javac -version on my machine is 1.8.0_181-google-v7 >>>> >>> >>>>>> >>>> >>> >>>>>> >>>> >>> >>>>>> I believe we can drop MAVEN_VERSION now that it is no longer >>>> used. I do not think it is needed to add a Gradle version either because >>>> the version itself is part of the repo anyway. >>>> >>> >>>>>> >>>> >>> >>>>>> I do not know if java, python etc. versions are helpful. >>>> Maybe others can comment. I would prefer to reduce the load on the release >>>> manager and drop this if this is not particularly important. >>>> >>> >>>>>> >>>> >>> >>>>>>> >>>> >>> >>>>>>> >>>> >>> >>>>>>> >>>> >>> >>>>>>> On Mon, Jan 13, 2020 at 7:37 PM Valentyn Tymofieiev < >>>> valen...@google.com> wrote: >>>> >>> >>>>>>>> >>>> >>> >>>>>>>> There are some issues in this message, part of the message >>>> is still a template (1.2.3, TODO, MAVEN_VERSION). >>>> >>> >>>>>>>> Before I noticed these issues, I ran a few Batch and >>>> Streaming Python 3.7 pipelines using Direct and Dataflow runners, and they >>>> all succeeded. >>>> >>> >>>>>>>> >>>> >>> >>>>>>>> On Mon, Jan 13, 2020 at 4:09 PM Udi Meiri < >>>> eh...@google.com> wrote: >>>> >>> >>>>>>>>> >>>> >>> >>>>>>>>> Hi everyone, >>>> >>> >>>>>>>>> Please review and vote on the release candidate #3 for >>>> the version 1.2.3, as follows: >>>> >>> >>>>>>>>> [ ] +1, Approve the release >>>> >>> >>>>>>>>> [ ] -1, Do not approve the release (please provide >>>> specific comments) >>>> >>> >>>>>>>>> >>>> >>> >>>>>>>>> >>>> >>> >>>>>>>>> The complete staging area is available for your review, >>>> which includes: >>>> >>> >>>>>>>>> * JIRA release notes [1], >>>> >>> >>>>>>>>> * the official Apache source release to be deployed to >>>> dist.apache.org [2], which is signed with the key with fingerprint >>>> 8961 F3EF 8E79 6688 4067 87CF 587B 049C 36DA AFE6 [3], >>>> >>> >>>>>>>>> * all artifacts to be deployed to the Maven Central >>>> Repository [4], >>>> >>> >>>>>>>>> * source code tag "v1.2.3-RC3" [5], >>>> >>> >>>>>> >>>> >>> >>>>>> >>>> >>> >>>>>> Tag is "v2.18.0-RC1". This is correct in the referenced link. >>>> >>> >>>>>> >>>> >>> >>>>>>>>> >>>> >>> >>>>>>>>> * website pull request listing the release [6], >>>> publishing the API reference manual [7], and the blog post [8]. >>>> >>> >>>>>>>>> * Java artifacts were built with Maven MAVEN_VERSION and >>>> OpenJDK/Oracle JDK JDK_VERSION. >>>> >>> >>>>>>>>> TODO: do these versions matter, and are they stamped into >>>> the artifacts? >>>> >>> >>>>>>>>> * Python artifacts are deployed along with the source >>>> release to the dist.apache.org [2]. >>>> >>> >>>>>>>>> * Validation sheet with a tab for 2.18.0 release to help >>>> with validation [9]. >>>> >>> >>>>>>>>> * Docker images published to Docker Hub [10]. >>>> >>> >>>>>>>>> >>>> >>> >>>>>>>>> The vote will be open for at least 72 hours. It is >>>> adopted by majority approval, with at least 3 PMC affirmative votes. >>>> >>> >>>>>>>>> >>>> >>> >>>>>>>>> Thanks, >>>> >>> >>>>>>>>> Release Manager >>>> >>> >>>>>>>>> >>>> >>> >>>>>>>>> [1] >>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12346383&projectId=12319527 >>>> >>> >>>>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.18.0/ >>>> >>> >>>>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS >>>> >>> >>>>>>>>> [4] >>>> https://repository.apache.org/content/repositories/orgapachebeam-1090/ >>>> >>> >>>>>>>>> [5] https://github.com/apache/beam/tree/v2.18.0-RC1 >>>> >>> >>>>>>>>> [6] https://github.com/apache/beam/pull/10574 >>>> >>> >>>>>>>>> [7] https://github.com/apache/beam-site/pull/595 >>>> >>> >>>>>>>>> [8] https://github.com/apache/beam/pull/10575 >>>> >>> >>>>>>>>> [9] >>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1178617819 >>>> >>> >>>>>>>>> [10] https://hub.docker.com/u/apachebeam >>>> >>>
