True. It is the use case of Repeatedly(ProcessingTime(n)) that I would guess to be the primary case of concern.
Kenn On Mon, Sep 27, 2021 at 2:38 PM Robert Bradshaw <rober...@google.com> wrote: > As I read the code, it's only pipelines that use a Repeatedly trigger > that wrap an already lossy trigger that are declared to be themselves > lossy. If I'm mistaken, I'll certainly reconsider my vote (and thanks > for bringing this up). > > On Mon, Sep 27, 2021 at 2:21 PM Kenneth Knowles <k...@apache.org> wrote: > > > > My concern is that the error message is incorrect and every user of > 2.33.0 may be educated wrong, or be worried about data loss in Beam, or > fail to find the blog post or CHANGES, etc. > > > > Kenn > > > > On Mon, Sep 27, 2021 at 2:16 PM Udi Meiri <eh...@google.com> wrote: > >> > >> I don't know how rare it is, but there is a flag documented in CHANGES > and blog post that reverts to the old behavior. > >> > >> On Mon, Sep 27, 2021 at 2:12 PM Kenneth Knowles <k...@apache.org> > wrote: > >>> > >>> I guess my vote is -0 since I don't have enough context on this issue. > A number of people with more awareness of how severe this is have voted +1 > so I will not try to block the release. > >>> > >>> Kenn > >>> > >>> On Mon, Sep 27, 2021 at 2:11 PM Kenneth Knowles <k...@apache.org> > wrote: > >>>> > >>>> I have to disagree with the other PMC members here, or at least dig > in to the question: every pipeline that uses a Repeatedly trigger at the > top level will be rejected. Is this so rare in Python that it is OK? > >>>> > >>>> Kenn > >>>> > >>>> On Mon, Sep 27, 2021 at 1:56 PM Robert Burke <r...@google.com> wrote: > >>>>> > >>>>> On it. Thanks! > >>>>> > >>>>> On Mon, Sep 27, 2021 at 1:18 PM Udi Meiri <eh...@google.com> wrote: > >>>>>> > >>>>>> Daniel/Robert, feel free to make changes to this PR: > https://github.com/apache/beam/pull/15543 > >>>>>> > >>>>>> On Mon, Sep 27, 2021 at 12:08 PM Daniel Oliveira < > danolive...@google.com> wrote: > >>>>>>> > >>>>>>> +1 > >>>>>>> > >>>>>>> I hadn't realized the pipelines were still finishing successfully. > I retried wordcount with that in mind and confirmed it finishes > successfully, so this isn't a blocker. > >>>>>>> > >>>>>>> Although maybe we should add this to the "Known Issues" because I > can easily see those messages being interpreted as a pipeline failure. > >>>>>>> > >>>>>>> On Sun, Sep 26, 2021 at 7:26 PM Robert Burke <rob...@frantil.com> > wrote: > >>>>>>>> > >>>>>>>> That's on the SDK side, and it just means the PCollection metrics > are being returned, buy not handled by the SDK. At present the pipeline > results only handle PTransform metrics. > >>>>>>>> > >>>>>>>> As such it's not a regression, as adding those is still under > development. > >>>>>>>> > >>>>>>>> On Sun, Sep 26, 2021, 7:02 PM Daniel Oliveira < > danolive...@google.com> wrote: > >>>>>>>>> > >>>>>>>>> I tried validating wordcount with the Go SDK on Flink. The > pipeline failed with a wall of errors like the following. I tried this on > Flink 1.11, 1.12, and 1.13 job servers built from source at the RC1 commit, > same errors on all of them. > >>>>>>>>> > >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: > urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" > payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n9"} > >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: > urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" > payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n10"} > >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: > urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" > payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n8"} > >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: > urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" > payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n7"} > >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: > urn:"beam:metric:sampled_byte_size:v1" > type:"beam:metrics:distribution_int64:v1" > payload:"\x8d%\xfa\xbf\x05\x04\xa8\x0c" labels:{key:"PCOLLECTION" > value:"n7"} > >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: > urn:"beam:metric:sampled_byte_size:v1" > type:"beam:metrics:distribution_int64:v1" payload:"\xb9\x05\xcf6\x05\x11" > labels:{key:"PCOLLECTION" value:"n9"} > >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: > urn:"beam:metric:sampled_byte_size:v1" > type:"beam:metrics:distribution_int64:v1" payload:"\xc5\x05\x8a1\x04\x12" > labels:{key:"PCOLLECTION" value:"n8"} > >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: > urn:"beam:metric:sampled_byte_size:v1" > type:"beam:metrics:distribution_int64:v1" > payload:"\x8d%\xb3\xa7\x07\x14\"" labels:{key:"PCOLLECTION" value:"n10"} > >>>>>>>>>> {...} > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> @Robert Burke I think you might know what's going on here. Is > this solvable with a cherry-pick and a new RC? > >>>>>>>>> > >>>>>>>>> On Fri, Sep 24, 2021 at 4:25 PM Ahmet Altay <al...@google.com> > wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Fri, Sep 24, 2021 at 4:21 PM Udi Meiri <eh...@google.com> > wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Fri, Sep 24, 2021 at 4:18 PM Ahmet Altay <al...@google.com> > wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Thank you for the update related to allow_unsafe_triggers. My > vote is still a +1. > >>>>>>>>>>>> > >>>>>>>>>>>> If we decide to move forward with this RC, could you please > include this bug in the known issues list under the changes.md for this > release? > >>>>>>>>>>> > >>>>>>>>>>> Yes, that's included in > https://github.com/apache/beam/pull/15543/files. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Thank you! > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Fri, Sep 24, 2021 at 3:56 PM Chamikara Jayalath < > chamik...@google.com> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> +1 (binding) > >>>>>>>>>>>>> Validated a few scenarios from the spreadsheet. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Cham > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Fri, Sep 24, 2021 at 3:07 PM Robert Bradshaw < > rober...@google.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> +1 (binding) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> All the artifacts and signatures look good. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I don't think the unsafe trigger check is severe enough to > block the release. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Fri, Sep 24, 2021 at 2:36 PM Udi Meiri <eh...@google.com> > wrote: > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > Hi everyone, we found a bug during testing. It has to do > with Python SDK's allow_unsafe_triggers check. > >>>>>>>>>>>>>> > There is a preliminary fix that will go to master. > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > For the 2.33.0 release, I'm leaning towards not making a > new RC since there is a workaround: pass the flag --allow_unsafe_triggers. > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > Please reevaluate your votes accordingly and recast if > you've changed your vote. > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > Thanks > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > On Fri, Sep 24, 2021 at 12:48 PM Alexey Romanenko < > aromanenko....@gmail.com> wrote: > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> On 24 Sep 2021, at 20:45, Udi Meiri <eh...@google.com> > wrote: > >>>>>>>>>>>>>> >> Alexey is this something that we should put in the > release notes, or some other change? > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> Yes, I think it could be helpful to mention that Beam > Jackson’s deps was bumped and it may require an update of Jackson’s runtime > deps for Spark 2 users as well. > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> — > >>>>>>>>>>>>>> >> Alexey > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> — > >>>>>>>>>>>>>> >>> Alexey > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> On 24 Sep 2021, at 16:17, Alexey Romanenko < > aromanenko....@gmail.com> wrote: > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> I checked with beam-samples [1] and noticed an issue to > run some pipelines with Spark 2 runner (Spark 3 seems is ok). > >>>>>>>>>>>>>> >>> It looks that it’s caused by new Jackson's version > updated recently [2], even if it’s a minor update but it works fine with a > previous one. > >>>>>>>>>>>>>> >>> I’ll try to find a workaround and get back with a > results of this. > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> [1] https://github.com/Talend/beam-samples/ > >>>>>>>>>>>>>> >>> [2] > https://github.com/apache/beam/commit/9694f70df1447e96684b665279679edafec13a0c > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> — > >>>>>>>>>>>>>> >>> Alexey > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> On 24 Sep 2021, at 11:17, Jan Lukavský <je...@seznam.cz> > wrote: > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> +1 (non-binding) > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> Validated several use-cases using non-portable Flink > with Java SDK. > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> On 9/24/21 4:55 AM, Valentyn Tymofieiev wrote: > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> +1. Ran several Python batch and streaming pipelines on > Dataflow and checked that Dataflow containers have required dependencies of > Beam. > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> On Thu, Sep 23, 2021 at 7:03 PM Robert Burke < > lostl...@apache.org> wrote: > >>>>>>>>>>>>>> >>>> > >>>>>>>>>>>>>> >>>> +1 (non-binding) > >>>>>>>>>>>>>> >>>> I validated the Go Quickstart (wordcount), and my ray > tracer against the Go Direct runner, Dataflow, and Spark (ensuring the rc1 > tagged container was used) and they executed successfully. > >>>>>>>>>>>>>> >>>> > >>>>>>>>>>>>>> >>>> I needed to manually synthesize a pseudo-version to > ensure I was using the tagged branch version > (v2.0.0-20210914211513-b358127f9859) instead of simply using v2.33.0-RC1. > >>>>>>>>>>>>>> >>>> > >>>>>>>>>>>>>> >>>> It either can't find the package with the right > tagged version, or it can't find the version. It's not clear to me what the > issue is, but it's not notionally a release blocker. I'll investigate > further once we have a full release, as it's probably some unspecified > behavior due to how we transitioned to Go Modules (which strongly > recommended doing a major version bump for such transitions, which seems a > bit excessive for Beam as a whole...). > >>>>>>>>>>>>>> >>>> > >>>>>>>>>>>>>> >>>> On 2021/09/23 03:59:18, Ahmet Altay <al...@google.com> > wrote: > >>>>>>>>>>>>>> >>>> > +1 on the RC. I validated python quick start > examples on direct runners. > >>>>>>>>>>>>>> >>>> > Thank you Udi. > >>>>>>>>>>>>>> >>>> > > >>>>>>>>>>>>>> >>>> > On Wed, Sep 22, 2021 at 2:20 PM Robert Burke < > rob...@frantil.com> wrote: > >>>>>>>>>>>>>> >>>> > > >>>>>>>>>>>>>> >>>> > > Just an FYI that intend to validate the Go SDK for > this release but can't > >>>>>>>>>>>>>> >>>> > > get to it until tomorrow (Thursday). I'm catching > up from a week of > >>>>>>>>>>>>>> >>>> > > vacation. Apologies for the inconvenience. > >>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>> >>>> > > On Tue, Sep 21, 2021, 10:59 AM Udi Meiri < > eh...@google.com> wrote: > >>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>> >>>> > >> I updated the affected and fixed version fields > for > >>>>>>>>>>>>>> >>>> > >> https://issues.apache.org/jira/browse/BEAM-12356. > >>>>>>>>>>>>>> >>>> > >> > >>>>>>>>>>>>>> >>>> > >> > >>>>>>>>>>>>>> >>>> > >> On Tue, Sep 21, 2021 at 10:48 AM Reuven Lax < > re...@google.com> wrote: > >>>>>>>>>>>>>> >>>> > >> > >>>>>>>>>>>>>> >>>> > >>> Unfortunate - I didn't realize that > >>>>>>>>>>>>>> >>>> > >>> https://github.com/apache/beam/pull/15480 > didn't make the cut. > >>>>>>>>>>>>>> >>>> > >>> > >>>>>>>>>>>>>> >>>> > >>> This bug was a regression in Beam 2.32.0, and is > blocking multiple users > >>>>>>>>>>>>>> >>>> > >>> from updating. > >>>>>>>>>>>>>> >>>> > >>> > >>>>>>>>>>>>>> >>>> > >>> On Tue, Sep 21, 2021 at 10:33 AM Udi Meiri < > u...@apache.org> wrote: > >>>>>>>>>>>>>> >>>> > >>> > >>>>>>>>>>>>>> >>>> > >>>> Hi everyone, > >>>>>>>>>>>>>> >>>> > >>>> Please review and vote on the release candidate > #1 for the version > >>>>>>>>>>>>>> >>>> > >>>> 2.33.0, as follows: > >>>>>>>>>>>>>> >>>> > >>>> [ ] +1, Approve the release > >>>>>>>>>>>>>> >>>> > >>>> [ ] -1, Do not approve the release (please > provide specific comments) > >>>>>>>>>>>>>> >>>> > >>>> > >>>>>>>>>>>>>> >>>> > >>>> > >>>>>>>>>>>>>> >>>> > >>>> Reviewers are encouraged to test their own use > cases with the release > >>>>>>>>>>>>>> >>>> > >>>> candidate, and vote +1 if > >>>>>>>>>>>>>> >>>> > >>>> no issues are found. > >>>>>>>>>>>>>> >>>> > >>>> > >>>>>>>>>>>>>> >>>> > >>>> The complete staging area is available for your > review, which includes: > >>>>>>>>>>>>>> >>>> > >>>> * JIRA release notes [1], > >>>>>>>>>>>>>> >>>> > >>>> * the official Apache source release to be > deployed to dist.apache.org > >>>>>>>>>>>>>> >>>> > >>>> [2], which is signed with the key with > fingerprint 587B049C36DAAFE6 [3], > >>>>>>>>>>>>>> >>>> > >>>> * all artifacts to be deployed to the Maven > Central Repository [4], > >>>>>>>>>>>>>> >>>> > >>>> * source code tag "v2.33.0-RC1" [5], > >>>>>>>>>>>>>> >>>> > >>>> * website pull request listing the release [6], > the blog post [6], and > >>>>>>>>>>>>>> >>>> > >>>> publishing the API reference manual [7]. > >>>>>>>>>>>>>> >>>> > >>>> * Java artifacts were built with Maven 3.6.3 > and OpenJDK 1.8.0_181. > >>>>>>>>>>>>>> >>>> > >>>> * Python artifacts are deployed along with the > source release to the > >>>>>>>>>>>>>> >>>> > >>>> dist.apache.org [2] and pypy[8]. > >>>>>>>>>>>>>> >>>> > >>>> * Validation sheet with a tab for 2.33.0 > release to help with > >>>>>>>>>>>>>> >>>> > >>>> validation [9]. > >>>>>>>>>>>>>> >>>> > >>>> * Docker images published to Docker Hub [10]. > >>>>>>>>>>>>>> >>>> > >>>> > >>>>>>>>>>>>>> >>>> > >>>> The vote will be open for at least 72 hours. It > is adopted by majority > >>>>>>>>>>>>>> >>>> > >>>> approval, with at least 3 PMC affirmative votes. > >>>>>>>>>>>>>> >>>> > >>>> > >>>>>>>>>>>>>> >>>> > >>>> For guidelines on how to try the release in > your projects, check out > >>>>>>>>>>>>>> >>>> > >>>> our blog post at > https://beam.apache.org/blog/validate-beam-release/. > >>>>>>>>>>>>>> >>>> > >>>> > >>>>>>>>>>>>>> >>>> > >>>> Thanks, > >>>>>>>>>>>>>> >>>> > >>>> Release Manager > >>>>>>>>>>>>>> >>>> > >>>> > >>>>>>>>>>>>>> >>>> > >>>> [1] > >>>>>>>>>>>>>> >>>> > >>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12350404 > >>>>>>>>>>>>>> >>>> > >>>> [2] > https://dist.apache.org/repos/dist/dev/beam/2.33.0/ > >>>>>>>>>>>>>> >>>> > >>>> [3] > https://dist.apache.org/repos/dist/release/beam/KEYS > >>>>>>>>>>>>>> >>>> > >>>> [4] > >>>>>>>>>>>>>> >>>> > >>>> > https://repository.apache.org/content/repositories/orgapachebeam-1234/ > >>>>>>>>>>>>>> >>>> > >>>> [5] > https://github.com/apache/beam/tree/v2.33.0-RC1 > >>>>>>>>>>>>>> >>>> > >>>> [6] https://github.com/apache/beam/pull/15543 > >>>>>>>>>>>>>> >>>> > >>>> [7] > https://github.com/apache/beam-site/pull/619 > >>>>>>>>>>>>>> >>>> > >>>> [8] > https://pypi.org/project/apache-beam/2.33.0rc1/ > >>>>>>>>>>>>>> >>>> > >>>> [9] > >>>>>>>>>>>>>> >>>> > >>>> > https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1705275493 > >>>>>>>>>>>>>> >>>> > >>>> [10] > https://hub.docker.com/search?q=apache%2Fbeam&type=image > >>>>>>>>>>>>>> >>>> > >>>> > >>>>>>>>>>>>>> >>>> > >>>> > >>>>>>>>>>>>>> >>>> > > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >>> > >>>>>>>>>>>>>> >> >
