I have to disagree with the other PMC members here, or at least dig in to the question: every pipeline that uses a Repeatedly trigger at the top level will be rejected. Is this so rare in Python that it is OK?
Kenn On Mon, Sep 27, 2021 at 1:56 PM Robert Burke <r...@google.com> wrote: > On it. Thanks! > > On Mon, Sep 27, 2021 at 1:18 PM Udi Meiri <eh...@google.com> wrote: > >> Daniel/Robert, feel free to make changes to this PR: >> https://github.com/apache/beam/pull/15543 >> >> On Mon, Sep 27, 2021 at 12:08 PM Daniel Oliveira <danolive...@google.com> >> wrote: >> >>> +1 >>> >>> I hadn't realized the pipelines were still finishing successfully. I >>> retried wordcount with that in mind and confirmed it finishes successfully, >>> so this isn't a blocker. >>> >>> Although maybe we should add this to the "Known Issues" because I can >>> easily see those messages being interpreted as a pipeline failure. >>> >>> On Sun, Sep 26, 2021 at 7:26 PM Robert Burke <rob...@frantil.com> wrote: >>> >>>> That's on the SDK side, and it just means the PCollection metrics are >>>> being returned, buy not handled by the SDK. At present the pipeline results >>>> only handle PTransform metrics. >>>> >>>> As such it's not a regression, as adding those is still under >>>> development. >>>> >>>> On Sun, Sep 26, 2021, 7:02 PM Daniel Oliveira <danolive...@google.com> >>>> wrote: >>>> >>>>> I tried validating wordcount with the Go SDK on Flink. The pipeline >>>>> failed with a wall of errors like the following. I tried this on Flink >>>>> 1.11, 1.12, and 1.13 job servers built from source at the RC1 commit, same >>>>> errors on all of them. >>>>> >>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" >>>>>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n9"} >>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" >>>>>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n10"} >>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" >>>>>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n8"} >>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" >>>>>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n7"} >>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>> urn:"beam:metric:sampled_byte_size:v1" >>>>>> type:"beam:metrics:distribution_int64:v1" >>>>>> payload:"\x8d%\xfa\xbf\x05\x04\xa8\x0c" labels:{key:"PCOLLECTION" >>>>>> value:"n7"} >>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>> urn:"beam:metric:sampled_byte_size:v1" >>>>>> type:"beam:metrics:distribution_int64:v1" >>>>>> payload:"\xb9\x05\xcf6\x05\x11" >>>>>> labels:{key:"PCOLLECTION" value:"n9"} >>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>> urn:"beam:metric:sampled_byte_size:v1" >>>>>> type:"beam:metrics:distribution_int64:v1" >>>>>> payload:"\xc5\x05\x8a1\x04\x12" >>>>>> labels:{key:"PCOLLECTION" value:"n8"} >>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>> urn:"beam:metric:sampled_byte_size:v1" >>>>>> type:"beam:metrics:distribution_int64:v1" >>>>>> payload:"\x8d%\xb3\xa7\x07\x14\"" labels:{key:"PCOLLECTION" >>>>>> value:"n10"} >>>>>> {...} >>>>> >>>>> >>>>> @Robert Burke <r...@google.com> I think you might know what's going >>>>> on here. Is this solvable with a cherry-pick and a new RC? >>>>> >>>>> On Fri, Sep 24, 2021 at 4:25 PM Ahmet Altay <al...@google.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Sep 24, 2021 at 4:21 PM Udi Meiri <eh...@google.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 24, 2021 at 4:18 PM Ahmet Altay <al...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thank you for the update related to allow_unsafe_triggers. My vote >>>>>>>> is still a +1. >>>>>>>> >>>>>>>> If we decide to move forward with this RC, could you please include >>>>>>>> this bug in the known issues list under the changes.md for this >>>>>>>> release? >>>>>>>> >>>>>>> Yes, that's included in >>>>>>> https://github.com/apache/beam/pull/15543/files. >>>>>>> >>>>>> >>>>>> Thank you! >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Fri, Sep 24, 2021 at 3:56 PM Chamikara Jayalath < >>>>>>>> chamik...@google.com> wrote: >>>>>>>> >>>>>>>>> +1 (binding) >>>>>>>>> Validated a few scenarios from the spreadsheet. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Cham >>>>>>>>> >>>>>>>>> On Fri, Sep 24, 2021 at 3:07 PM Robert Bradshaw < >>>>>>>>> rober...@google.com> wrote: >>>>>>>>> >>>>>>>>>> +1 (binding) >>>>>>>>>> >>>>>>>>>> All the artifacts and signatures look good. >>>>>>>>>> >>>>>>>>>> I don't think the unsafe trigger check is severe enough to block >>>>>>>>>> the release. >>>>>>>>>> >>>>>>>>>> On Fri, Sep 24, 2021 at 2:36 PM Udi Meiri <eh...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> > >>>>>>>>>> > Hi everyone, we found a bug during testing. It has to do with >>>>>>>>>> Python SDK's allow_unsafe_triggers check. >>>>>>>>>> > There is a preliminary fix that will go to master. >>>>>>>>>> > >>>>>>>>>> > For the 2.33.0 release, I'm leaning towards not making a new RC >>>>>>>>>> since there is a workaround: pass the flag --allow_unsafe_triggers. >>>>>>>>>> > >>>>>>>>>> > Please reevaluate your votes accordingly and recast if you've >>>>>>>>>> changed your vote. >>>>>>>>>> > >>>>>>>>>> > Thanks >>>>>>>>>> > >>>>>>>>>> > On Fri, Sep 24, 2021 at 12:48 PM Alexey Romanenko < >>>>>>>>>> aromanenko....@gmail.com> wrote: >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> On 24 Sep 2021, at 20:45, Udi Meiri <eh...@google.com> wrote: >>>>>>>>>> >> Alexey is this something that we should put in the release >>>>>>>>>> notes, or some other change? >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> Yes, I think it could be helpful to mention that Beam >>>>>>>>>> Jackson’s deps was bumped and it may require an update of Jackson’s >>>>>>>>>> runtime >>>>>>>>>> deps for Spark 2 users as well. >>>>>>>>>> >> >>>>>>>>>> >> — >>>>>>>>>> >> Alexey >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> — >>>>>>>>>> >>> Alexey >>>>>>>>>> >>> >>>>>>>>>> >>> On 24 Sep 2021, at 16:17, Alexey Romanenko < >>>>>>>>>> aromanenko....@gmail.com> wrote: >>>>>>>>>> >>> >>>>>>>>>> >>> I checked with beam-samples [1] and noticed an issue to run >>>>>>>>>> some pipelines with Spark 2 runner (Spark 3 seems is ok). >>>>>>>>>> >>> It looks that it’s caused by new Jackson's version updated >>>>>>>>>> recently [2], even if it’s a minor update but it works fine with a >>>>>>>>>> previous >>>>>>>>>> one. >>>>>>>>>> >>> I’ll try to find a workaround and get back with a results of >>>>>>>>>> this. >>>>>>>>>> >>> >>>>>>>>>> >>> [1] https://github.com/Talend/beam-samples/ >>>>>>>>>> >>> [2] >>>>>>>>>> https://github.com/apache/beam/commit/9694f70df1447e96684b665279679edafec13a0c >>>>>>>>>> >>> >>>>>>>>>> >>> — >>>>>>>>>> >>> Alexey >>>>>>>>>> >>> >>>>>>>>>> >>> On 24 Sep 2021, at 11:17, Jan Lukavský <je...@seznam.cz> >>>>>>>>>> wrote: >>>>>>>>>> >>> >>>>>>>>>> >>> +1 (non-binding) >>>>>>>>>> >>> >>>>>>>>>> >>> Validated several use-cases using non-portable Flink with >>>>>>>>>> Java SDK. >>>>>>>>>> >>> >>>>>>>>>> >>> On 9/24/21 4:55 AM, Valentyn Tymofieiev wrote: >>>>>>>>>> >>> >>>>>>>>>> >>> +1. Ran several Python batch and streaming pipelines on >>>>>>>>>> Dataflow and checked that Dataflow containers have required >>>>>>>>>> dependencies of >>>>>>>>>> Beam. >>>>>>>>>> >>> >>>>>>>>>> >>> On Thu, Sep 23, 2021 at 7:03 PM Robert Burke < >>>>>>>>>> lostl...@apache.org> wrote: >>>>>>>>>> >>>> >>>>>>>>>> >>>> +1 (non-binding) >>>>>>>>>> >>>> I validated the Go Quickstart (wordcount), and my ray tracer >>>>>>>>>> against the Go Direct runner, Dataflow, and Spark (ensuring the rc1 >>>>>>>>>> tagged >>>>>>>>>> container was used) and they executed successfully. >>>>>>>>>> >>>> >>>>>>>>>> >>>> I needed to manually synthesize a pseudo-version to ensure I >>>>>>>>>> was using the tagged branch version >>>>>>>>>> (v2.0.0-20210914211513-b358127f9859) >>>>>>>>>> instead of simply using v2.33.0-RC1. >>>>>>>>>> >>>> >>>>>>>>>> >>>> It either can't find the package with the right tagged >>>>>>>>>> version, or it can't find the version. It's not clear to me what the >>>>>>>>>> issue >>>>>>>>>> is, but it's not notionally a release blocker. I'll investigate >>>>>>>>>> further >>>>>>>>>> once we have a full release, as it's probably some unspecified >>>>>>>>>> behavior due >>>>>>>>>> to how we transitioned to Go Modules (which strongly recommended >>>>>>>>>> doing a >>>>>>>>>> major version bump for such transitions, which seems a bit excessive >>>>>>>>>> for >>>>>>>>>> Beam as a whole...). >>>>>>>>>> >>>> >>>>>>>>>> >>>> On 2021/09/23 03:59:18, Ahmet Altay <al...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>> > +1 on the RC. I validated python quick start examples on >>>>>>>>>> direct runners. >>>>>>>>>> >>>> > Thank you Udi. >>>>>>>>>> >>>> > >>>>>>>>>> >>>> > On Wed, Sep 22, 2021 at 2:20 PM Robert Burke < >>>>>>>>>> rob...@frantil.com> wrote: >>>>>>>>>> >>>> > >>>>>>>>>> >>>> > > Just an FYI that intend to validate the Go SDK for this >>>>>>>>>> release but can't >>>>>>>>>> >>>> > > get to it until tomorrow (Thursday). I'm catching up >>>>>>>>>> from a week of >>>>>>>>>> >>>> > > vacation. Apologies for the inconvenience. >>>>>>>>>> >>>> > > >>>>>>>>>> >>>> > > On Tue, Sep 21, 2021, 10:59 AM Udi Meiri < >>>>>>>>>> eh...@google.com> wrote: >>>>>>>>>> >>>> > > >>>>>>>>>> >>>> > >> I updated the affected and fixed version fields for >>>>>>>>>> >>>> > >> https://issues.apache.org/jira/browse/BEAM-12356. >>>>>>>>>> >>>> > >> >>>>>>>>>> >>>> > >> >>>>>>>>>> >>>> > >> On Tue, Sep 21, 2021 at 10:48 AM Reuven Lax < >>>>>>>>>> re...@google.com> wrote: >>>>>>>>>> >>>> > >> >>>>>>>>>> >>>> > >>> Unfortunate - I didn't realize that >>>>>>>>>> >>>> > >>> https://github.com/apache/beam/pull/15480 didn't make >>>>>>>>>> the cut. >>>>>>>>>> >>>> > >>> >>>>>>>>>> >>>> > >>> This bug was a regression in Beam 2.32.0, and is >>>>>>>>>> blocking multiple users >>>>>>>>>> >>>> > >>> from updating. >>>>>>>>>> >>>> > >>> >>>>>>>>>> >>>> > >>> On Tue, Sep 21, 2021 at 10:33 AM Udi Meiri < >>>>>>>>>> u...@apache.org> wrote: >>>>>>>>>> >>>> > >>> >>>>>>>>>> >>>> > >>>> Hi everyone, >>>>>>>>>> >>>> > >>>> Please review and vote on the release candidate #1 >>>>>>>>>> for the version >>>>>>>>>> >>>> > >>>> 2.33.0, as follows: >>>>>>>>>> >>>> > >>>> [ ] +1, Approve the release >>>>>>>>>> >>>> > >>>> [ ] -1, Do not approve the release (please provide >>>>>>>>>> specific comments) >>>>>>>>>> >>>> > >>>> >>>>>>>>>> >>>> > >>>> >>>>>>>>>> >>>> > >>>> Reviewers are encouraged to test their own use cases >>>>>>>>>> with the release >>>>>>>>>> >>>> > >>>> candidate, and vote +1 if >>>>>>>>>> >>>> > >>>> no issues are found. >>>>>>>>>> >>>> > >>>> >>>>>>>>>> >>>> > >>>> The complete staging area is available for your >>>>>>>>>> review, which includes: >>>>>>>>>> >>>> > >>>> * JIRA release notes [1], >>>>>>>>>> >>>> > >>>> * the official Apache source release to be deployed >>>>>>>>>> to dist.apache.org >>>>>>>>>> >>>> > >>>> [2], which is signed with the key with fingerprint >>>>>>>>>> 587B049C36DAAFE6 [3], >>>>>>>>>> >>>> > >>>> * all artifacts to be deployed to the Maven Central >>>>>>>>>> Repository [4], >>>>>>>>>> >>>> > >>>> * source code tag "v2.33.0-RC1" [5], >>>>>>>>>> >>>> > >>>> * website pull request listing the release [6], the >>>>>>>>>> blog post [6], and >>>>>>>>>> >>>> > >>>> publishing the API reference manual [7]. >>>>>>>>>> >>>> > >>>> * Java artifacts were built with Maven 3.6.3 and >>>>>>>>>> OpenJDK 1.8.0_181. >>>>>>>>>> >>>> > >>>> * Python artifacts are deployed along with the source >>>>>>>>>> release to the >>>>>>>>>> >>>> > >>>> dist.apache.org [2] and pypy[8]. >>>>>>>>>> >>>> > >>>> * Validation sheet with a tab for 2.33.0 release to >>>>>>>>>> help with >>>>>>>>>> >>>> > >>>> validation [9]. >>>>>>>>>> >>>> > >>>> * Docker images published to Docker Hub [10]. >>>>>>>>>> >>>> > >>>> >>>>>>>>>> >>>> > >>>> The vote will be open for at least 72 hours. It is >>>>>>>>>> adopted by majority >>>>>>>>>> >>>> > >>>> approval, with at least 3 PMC affirmative votes. >>>>>>>>>> >>>> > >>>> >>>>>>>>>> >>>> > >>>> For guidelines on how to try the release in your >>>>>>>>>> projects, check out >>>>>>>>>> >>>> > >>>> our blog post at >>>>>>>>>> https://beam.apache.org/blog/validate-beam-release/. >>>>>>>>>> >>>> > >>>> >>>>>>>>>> >>>> > >>>> Thanks, >>>>>>>>>> >>>> > >>>> Release Manager >>>>>>>>>> >>>> > >>>> >>>>>>>>>> >>>> > >>>> [1] >>>>>>>>>> >>>> > >>>> >>>>>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12350404 >>>>>>>>>> >>>> > >>>> [2] >>>>>>>>>> https://dist.apache.org/repos/dist/dev/beam/2.33.0/ >>>>>>>>>> >>>> > >>>> [3] >>>>>>>>>> https://dist.apache.org/repos/dist/release/beam/KEYS >>>>>>>>>> >>>> > >>>> [4] >>>>>>>>>> >>>> > >>>> >>>>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1234/ >>>>>>>>>> >>>> > >>>> [5] https://github.com/apache/beam/tree/v2.33.0-RC1 >>>>>>>>>> >>>> > >>>> [6] https://github.com/apache/beam/pull/15543 >>>>>>>>>> >>>> > >>>> [7] https://github.com/apache/beam-site/pull/619 >>>>>>>>>> >>>> > >>>> [8] https://pypi.org/project/apache-beam/2.33.0rc1/ >>>>>>>>>> >>>> > >>>> [9] >>>>>>>>>> >>>> > >>>> >>>>>>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1705275493 >>>>>>>>>> >>>> > >>>> [10] >>>>>>>>>> https://hub.docker.com/search?q=apache%2Fbeam&type=image >>>>>>>>>> >>>> > >>>> >>>>>>>>>> >>>> > >>>> >>>>>>>>>> >>>> > >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>
