> I am not sure what's the status of this work - should this block the release until we can get those containers out?
We can release Flink 1.11 and Flink 1.12 containers and I'm *pretty sure* they would work correctly, but to be totally sure we'd have to upgrade our portable Flink tests to cover the new containers. Rather than rush new portable Flink versions into the release, I think it may be better to document that they aren't officially supported for now (users can always build the job server containers manually if they need). The larger problem here is that we need to improve the process of adding new Flink versions. Specifically, we need a single source (or at least fewer sources) of truth for supported Flink versions so that we don't have things getting out of sync again. https://issues.apache.org/jira/browse/BEAM-9541 On Mon, Dec 28, 2020 at 10:31 AM Pablo Estrada <pabl...@google.com> wrote: > As an update on the release, I've been made aware of, and merged all > cherry picks. I was in the process of building rc2, but I'm pausing to > resolve the issue with these containers. > Best > -P. > > On Mon, Dec 28, 2020, 1:11 PM Pablo Estrada <pabl...@google.com> wrote: > >> Good catch Ismael. Thanks! >> >> I've created https://issues.apache.org/jira/browse/INFRA-21238 to >> request the repositories to be created. >> >> I am not sure what's the status of this work - should this block the >> release until we can get those containers out? >> cc: @Kyle Weaver <kcwea...@google.com> >> Best >> -P. >> >> On Mon, Dec 28, 2020 at 9:48 AM Ismaël Mejía <ieme...@gmail.com> wrote: >> >>> It seems we are not publishing the latest versions of the Flink Job >>> Server (Flink 1.11 and 1.12) docker images >>> >>> These do not exist: >>> >>> https://hub.docker.com/r/apache/beam_flink1.11_job_server >>> https://hub.docker.com/r/apache/beam_flink1.12_job_server >>> >>> but this does and has the good 2.27.0_rc1 tag: >>> https://hub.docker.com/r/apache/beam_flink1.10_job_server >>> >>> I wonder if the issue might be related to the fact that we need to >>> request the repo to be created or if it is something different. >>> >>> On Thu, Dec 24, 2020 at 5:33 PM Brian Hulette <bhule...@google.com> >>> wrote: >>> > >>> > +Boyuan Zhang helped me get to the bottom of the sql_taxi issue. The >>> problem is with the WriteStringsToPubSub API, which is deprecated since >>> 2.7.0, but used in the example. Boyuan has [1] out to fix >>> WriteStringsToPubSub and I just sent [2] to replace WriteStringsToPubSub >>> with WriteToPubSub in example code. Issue is tracked in [3]. >>> > >>> > [1] https://github.com/apache/beam/pull/13614 >>> > [2] https://github.com/apache/beam/pull/13615 >>> > [3] https://issues.apache.org/jira/browse/BEAM-11524 >>> > >>> > On Thu, Dec 24, 2020 at 8:26 AM Pablo Estrada <pabl...@google.com> >>> wrote: >>> >> >>> >> Alright! Thanks everyone for your validations. I'm cancelling this >>> RC, and I'll perform cherry picks to prepare the next one. >>> >> >>> >> Please update this thread with any other cherry pick requests! >>> >> -P. >>> >> >>> >> On Thu, Dec 24, 2020, 3:17 AM Ismaël Mejía <ieme...@gmail.com> wrote: >>> >>> >>> >>> It might be a good idea to include also: >>> >>> >>> >>> [BEAM-11403] Cache UnboundedReader per UnboundedSourceRestriction in >>> >>> SDF Wrapper DoFn >>> >>> https://github.com/apache/beam/pull/13592 >>> >>> >>> >>> So Java development experience is less affected (as with 2.26.0) >>> (There >>> >>> is a flag to exclude but defaults matter). >>> >>> >>> >>> On Thu, Dec 24, 2020 at 2:56 AM Valentyn Tymofieiev < >>> valen...@google.com> wrote: >>> >>> > >>> >>> > We discovered a regression on CombineFn.from_callable() started in >>> 2.26.0. Even though it's not a regression in 2.27.0, I strongly prefer we >>> fix it in 2.27.0 as it leads to buggy behavior, so I vote -1. >>> >>> > >>> >>> > The fix to release branch is in flight: >>> https://github.com/apache/beam/pull/13613. >>> >>> > >>> >>> > >>> >>> > >>> >>> > On Wed, Dec 23, 2020 at 3:38 PM Brian Hulette <bhule...@google.com> >>> wrote: >>> >>> >> >>> >>> >> -1 (non-binding) >>> >>> >> Good news: I validated a dataframe pipeline on Dataflow which >>> looked good (with expected performance improvements!) >>> >>> >> Bad news: I also tried to run the sql_taxi example pipeline >>> (streaming SQL in python) on Dataflow and ran into PubSub IO related >>> issues. The example fails in the same way with 2.26.0, but it works in >>> 2.25.0. It's possible this is a Dataflow bug and not a Beam one, but I'd >>> like to investigate further to make sure. >>> >>> >> >>> >>> >> On Wed, Dec 23, 2020 at 12:25 PM Kyle Weaver <kcwea...@google.com> >>> wrote: >>> >>> >>> >>> >>> >>> +1 (non-binding) Validated wordcount with Python source + Flink >>> and Spark job server jars. Also checked that the ...:sql:udf jar was added >>> and includes our cherry-picks. Thanks Pablo :) >>> >>> >>> >>> >>> >>> On Wed, Dec 23, 2020 at 12:02 PM Ahmet Altay <al...@google.com> >>> wrote: >>> >>> >>>> >>> >>> >>>> +1 (binding). >>> >>> >>>> >>> >>> >>>> I validated python quickstarts. Thank you Pablo. >>> >>> >>>> >>> >>> >>>> On Tue, Dec 22, 2020 at 10:04 PM Jean-Baptiste Onofre < >>> j...@nanthrax.net> wrote: >>> >>> >>>>> >>> >>> >>>>> +1 (binding) >>> >>> >>>>> >>> >>> >>>>> Regards >>> >>> >>>>> JB >>> >>> >>>>> >>> >>> >>>>> Le 23 déc. 2020 à 06:46, Pablo Estrada <pabl...@google.com> a >>> écrit : >>> >>> >>>>> >>> >>> >>>>> Hi everyone, >>> >>> >>>>> Please review and vote on the release candidate #1 for the >>> version 2.27.0, as follows: >>> >>> >>>>> [ ] +1, Approve the release >>> >>> >>>>> [ ] -1, Do not approve the release (please provide specific >>> comments) >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> Reviewers are encouraged to test their own use cases with the >>> release candidate, and vote +1 >>> >>> >>>>> if no issues are found. >>> >>> >>>>> >>> >>> >>>>> The complete staging area is available for your review, which >>> includes: >>> >>> >>>>> * JIRA release notes [1], >>> >>> >>>>> * the official Apache source release to be deployed to >>> dist.apache.org [2], which is signed with the key with fingerprint >>> C79DDD47DAF3808F0B9DDFAC02B2D9F742008494 [3], >>> >>> >>>>> * all artifacts to be deployed to the Maven Central Repository >>> [4], >>> >>> >>>>> * source code tag "v2.27.0-RC1" [5], >>> >>> >>>>> * website pull request listing the release [6], publishing the >>> API reference manual [7], and the blog post [8]. >>> >>> >>>>> * Python artifacts are deployed along with the source release >>> to the dist.apache.org [2]. >>> >>> >>>>> * Validation sheet with a tab for 2.27.0 release to help with >>> validation [9]. >>> >>> >>>>> * Docker images published to Docker Hub [10]. >>> >>> >>>>> >>> >>> >>>>> The vote will be open for at least 72 hours, but given the >>> holidays, we will likely extend for a few more days. The release will be >>> adopted by majority approval, with at least 3 PMC affirmative votes. >>> >>> >>>>> >>> >>> >>>>> Thanks, >>> >>> >>>>> -P. >>> >>> >>>>> >>> >>> >>>>> [1] >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12349380 >>> >>> >>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.27.0/ >>> >>> >>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS >>> >>> >>>>> [4] >>> https://repository.apache.org/content/repositories/orgapachebeam-1145/ >>> >>> >>>>> [5] https://github.com/apache/beam/tree/v2.27.0-RC1 >>> >>> >>>>> [6] https://github.com/apache/beam/pull/13602 >>> >>> >>>>> [7] https://github.com/apache/beam-site/pull/610 >>> >>> >>>>> [8] https://github.com/apache/beam/pull/13603 >>> >>> >>>>> [9] >>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=194829106 >>> >>> >>>>> [10] https://hub.docker.com/search?q=apache%2Fbeam&type=image >>> >>> >>>>> >>> >>> >>>>> >>> >>
