Re: [RESULT] [VOTE] Release 2.33.0, release candidate #1

[Udi Meiri]

I spoke too soon. We will be doing an rc2

On Mon, Sep 27, 2021 at 1:29 PM Udi Meiri  wrote:

> I'm happy to announce that we have unanimously approved this release.
>
> There are 8 approving votes, 4 of which are binding:
> * Ahmet Altay
> * Alexey Romanenko
> * Robert Bradshaw
> * Chamikara Jayalath
>
> There are no disapproving votes.
>
> Thanks everyone!
>
>

Re: [VOTE] Release 2.33.0, release candidate 1

[Kenneth Knowles]

Ah, thanks for the correction. AfterProcessingTime and AfterCount should
both be flagged, as should AfterEvery, and AfterAny. AfterEach is
different, since if any of the sequenced triggers never finish, the whole
thing will never finish.

Kenn

On Mon, Sep 27, 2021 at 3:51 PM Zachary Houfek  wrote:

> > This was discovered because Repeatedly(AfterProcessingTime(n))
>
> Wasn't it Repeatedly(AfterCount(n)) that caused the problem? At least
> that's what caused the failing test that prompted the fix.
>
> With that said, is the concern that AfterProcessingTime *should* include
> the CONDITION_NOT_GUARANTEED flag? It isn't right now, so I'm checking.
>
> Another thing I haven't seen mentioned: `Repeatedly` is or can be used in
> other triggers like AfterWatermark (because of the late trigger),
> AfterEach, AfterEvery, and AfterAny, so the bug will propagate to those
> triggers as well. Two of these actually had to have their tests updated
> in the fix
> 
> .
>
> Regards,
> Zach
>
>
>
> On Mon, Sep 27, 2021 at 6:30 PM Robert Bradshaw 
> wrote:
>
>> $ git checkout release-2.33.0
>> Switched to branch 'release-2.33.0'
>> Your branch is up to date with 'github/release-2.33.0'.
>>
>> $ python -c 'import apache_beam as beam;
>>
>> print(beam.transforms.trigger.Repeatedly(beam.transforms.trigger.AfterProcessingTime(5)).may_lose_data(None))'
>> DataLossReason.NO_POTENTIAL_LOSS
>>
>> The MAY_FINISH and CONDITION_NOT_GUARANTEED flags are disjoint.
>>
>> On Mon, Sep 27, 2021 at 3:15 PM Kenneth Knowles  wrote:
>> >
>> > This was discovered because Repeatedly(AfterProcessingTime(n)) does
>> require the workaround, because AfterProcessingTime(n) does terminate and
>> that is propagated by the Repeatedly implementation.
>> >
>> > Kenn
>> >
>> > On Mon, Sep 27, 2021 at 2:49 PM Robert Bradshaw 
>> wrote:
>> >>
>> >> On Mon, Sep 27, 2021 at 2:42 PM Kenneth Knowles 
>> wrote:
>> >> >
>> >> > True. It is the use case of Repeatedly(ProcessingTime(n)) that I
>> would guess to be the primary case of concern.
>> >>
>> >> This shouldn't require the workaround, right?
>> >>
>> >> If you're not comfortable with this, I'd say let's patch and re-cut.
>> >>
>> >>
>> >> > On Mon, Sep 27, 2021 at 2:38 PM Robert Bradshaw 
>> wrote:
>> >> >>
>> >> >> As I read the code, it's only pipelines that use a Repeatedly
>> trigger
>> >> >> that wrap an already lossy trigger that are declared to be
>> themselves
>> >> >> lossy. If I'm mistaken, I'll certainly reconsider my vote (and
>> thanks
>> >> >> for bringing this up).
>> >> >>
>> >> >> On Mon, Sep 27, 2021 at 2:21 PM Kenneth Knowles 
>> wrote:
>> >> >> >
>> >> >> > My concern is that the error message is incorrect and every user
>> of 2.33.0 may be educated wrong, or be worried about data loss in Beam, or
>> fail to find the blog post or CHANGES, etc.
>> >> >> >
>> >> >> > Kenn
>> >> >> >
>> >> >> > On Mon, Sep 27, 2021 at 2:16 PM Udi Meiri 
>> wrote:
>> >> >> >>
>> >> >> >> I don't know how rare it is, but there is a flag documented in
>> CHANGES and blog post that reverts to the old behavior.
>> >> >> >>
>> >> >> >> On Mon, Sep 27, 2021 at 2:12 PM Kenneth Knowles 
>> wrote:
>> >> >> >>>
>> >> >> >>> I guess my vote is -0 since I don't have enough context on this
>> issue. A number of people with more awareness of how severe this is have
>> voted +1 so I will not try to block the release.
>> >> >> >>>
>> >> >> >>> Kenn
>> >> >> >>>
>> >> >> >>> On Mon, Sep 27, 2021 at 2:11 PM Kenneth Knowles <
>> k...@apache.org> wrote:
>> >> >> 
>> >> >>  I have to disagree with the other PMC members here, or at
>> least dig in to the question: every pipeline that uses a Repeatedly trigger
>> at the top level will be rejected. Is this so rare in Python that it is OK?
>> >> >> 
>> >> >>  Kenn
>> >> >> 
>> >> >>  On Mon, Sep 27, 2021 at 1:56 PM Robert Burke 
>> wrote:
>> >> >> >
>> >> >> > On it. Thanks!
>> >> >> >
>> >> >> > On Mon, Sep 27, 2021 at 1:18 PM Udi Meiri 
>> wrote:
>> >> >> >>
>> >> >> >> Daniel/Robert, feel free to make changes to this PR:
>> https://github.com/apache/beam/pull/15543
>> >> >> >>
>> >> >> >> On Mon, Sep 27, 2021 at 12:08 PM Daniel Oliveira <
>> danolive...@google.com> wrote:
>> >> >> >>>
>> >> >> >>> +1
>> >> >> >>>
>> >> >> >>> I hadn't realized the pipelines were still finishing
>> successfully. I retried wordcount with that in mind and confirmed it
>> finishes successfully, so this isn't a blocker.
>> >> >> >>>
>> >> >> >>> Although maybe we should add this to the "Known Issues"
>> because I can easily see those messages being interpreted as a pipeline
>> failure.
>> >> >> >>>
>> >> >> >>> On Sun, Sep 26, 2021 at 7:26 PM Robert Burke <
>> rob...@frantil.com> wrote:
>> >> >> 
>> >> >>  That's on the SDK side, and it just means the 

Re: [VOTE] Release 2.33.0, release candidate 1

[Kenneth Knowles]

This was discovered because Repeatedly(AfterProcessingTime(n)) does require
the workaround, because AfterProcessingTime(n) does terminate and that is
propagated by the Repeatedly implementation.

Kenn

On Mon, Sep 27, 2021 at 2:49 PM Robert Bradshaw  wrote:

> On Mon, Sep 27, 2021 at 2:42 PM Kenneth Knowles  wrote:
> >
> > True. It is the use case of Repeatedly(ProcessingTime(n)) that I would
> guess to be the primary case of concern.
>
> This shouldn't require the workaround, right?
>
> If you're not comfortable with this, I'd say let's patch and re-cut.
>
>
> > On Mon, Sep 27, 2021 at 2:38 PM Robert Bradshaw 
> wrote:
> >>
> >> As I read the code, it's only pipelines that use a Repeatedly trigger
> >> that wrap an already lossy trigger that are declared to be themselves
> >> lossy. If I'm mistaken, I'll certainly reconsider my vote (and thanks
> >> for bringing this up).
> >>
> >> On Mon, Sep 27, 2021 at 2:21 PM Kenneth Knowles 
> wrote:
> >> >
> >> > My concern is that the error message is incorrect and every user of
> 2.33.0 may be educated wrong, or be worried about data loss in Beam, or
> fail to find the blog post or CHANGES, etc.
> >> >
> >> > Kenn
> >> >
> >> > On Mon, Sep 27, 2021 at 2:16 PM Udi Meiri  wrote:
> >> >>
> >> >> I don't know how rare it is, but there is a flag documented in
> CHANGES and blog post that reverts to the old behavior.
> >> >>
> >> >> On Mon, Sep 27, 2021 at 2:12 PM Kenneth Knowles 
> wrote:
> >> >>>
> >> >>> I guess my vote is -0 since I don't have enough context on this
> issue. A number of people with more awareness of how severe this is have
> voted +1 so I will not try to block the release.
> >> >>>
> >> >>> Kenn
> >> >>>
> >> >>> On Mon, Sep 27, 2021 at 2:11 PM Kenneth Knowles 
> wrote:
> >> 
> >>  I have to disagree with the other PMC members here, or at least
> dig in to the question: every pipeline that uses a Repeatedly trigger at
> the top level will be rejected. Is this so rare in Python that it is OK?
> >> 
> >>  Kenn
> >> 
> >>  On Mon, Sep 27, 2021 at 1:56 PM Robert Burke 
> wrote:
> >> >
> >> > On it. Thanks!
> >> >
> >> > On Mon, Sep 27, 2021 at 1:18 PM Udi Meiri 
> wrote:
> >> >>
> >> >> Daniel/Robert, feel free to make changes to this PR:
> https://github.com/apache/beam/pull/15543
> >> >>
> >> >> On Mon, Sep 27, 2021 at 12:08 PM Daniel Oliveira <
> danolive...@google.com> wrote:
> >> >>>
> >> >>> +1
> >> >>>
> >> >>> I hadn't realized the pipelines were still finishing
> successfully. I retried wordcount with that in mind and confirmed it
> finishes successfully, so this isn't a blocker.
> >> >>>
> >> >>> Although maybe we should add this to the "Known Issues" because
> I can easily see those messages being interpreted as a pipeline failure.
> >> >>>
> >> >>> On Sun, Sep 26, 2021 at 7:26 PM Robert Burke <
> rob...@frantil.com> wrote:
> >> 
> >>  That's on the SDK side, and it just means the PCollection
> metrics are being returned, buy not handled by the SDK. At present the
> pipeline results only handle PTransform metrics.
> >> 
> >>  As such it's not a regression, as adding those is still under
> development.
> >> 
> >>  On Sun, Sep 26, 2021, 7:02 PM Daniel Oliveira <
> danolive...@google.com> wrote:
> >> >
> >> > I tried validating wordcount with the Go SDK on Flink. The
> pipeline failed with a wall of errors like the following. I tried this on
> Flink 1.11, 1.12, and 1.13 job servers built from source at the RC1 commit,
> same errors on all of them.
> >> >
> >> >> 2021/09/26 18:21:49 Failed to deduce Step from
> MonitoringInfo: urn:"beam:metric:element_count:v1"
> type:"beam:metrics:sum_int64:v1"  payload:"\x8d%"
> labels:{key:"PCOLLECTION"  value:"n9"}
> >> >> 2021/09/26 18:21:49 Failed to deduce Step from
> MonitoringInfo: urn:"beam:metric:element_count:v1"
> type:"beam:metrics:sum_int64:v1"  payload:"\x8d%"
> labels:{key:"PCOLLECTION"  value:"n10"}
> >> >> 2021/09/26 18:21:49 Failed to deduce Step from
> MonitoringInfo: urn:"beam:metric:element_count:v1"
> type:"beam:metrics:sum_int64:v1"  payload:"\x8d%"
> labels:{key:"PCOLLECTION"  value:"n8"}
> >> >> 2021/09/26 18:21:49 Failed to deduce Step from
> MonitoringInfo: urn:"beam:metric:element_count:v1"
> type:"beam:metrics:sum_int64:v1"  payload:"\x8d%"
> labels:{key:"PCOLLECTION"  value:"n7"}
> >> >> 2021/09/26 18:21:49 Failed to deduce Step from
> MonitoringInfo: urn:"beam:metric:sampled_byte_size:v1"
> type:"beam:metrics:distribution_int64:v1"
> payload:"\x8d%\xfa\xbf\x05\x04\xa8\x0c"  labels:{key:"PCOLLECTION"
> value:"n7"}
> >> >> 2021/09/26 18:21:49 Failed to deduce Step from
> MonitoringInfo: urn:"beam:metric:sampled_byte_size:v1"
> type:"beam:metrics:distribution_int64:v1"  payload:"\xb9\x05\xcf6\x05\x11"
> labels:{key:"PCOLLECTION"  value:"n9"}
> >> >> 2021/09/26 18:21:49 

Re: [VOTE] Release 2.33.0, release candidate 1

[Kenneth Knowles]

True. It is the use case of Repeatedly(ProcessingTime(n)) that I would
guess to be the primary case of concern.

Kenn

On Mon, Sep 27, 2021 at 2:38 PM Robert Bradshaw  wrote:

> As I read the code, it's only pipelines that use a Repeatedly trigger
> that wrap an already lossy trigger that are declared to be themselves
> lossy. If I'm mistaken, I'll certainly reconsider my vote (and thanks
> for bringing this up).
>
> On Mon, Sep 27, 2021 at 2:21 PM Kenneth Knowles  wrote:
> >
> > My concern is that the error message is incorrect and every user of
> 2.33.0 may be educated wrong, or be worried about data loss in Beam, or
> fail to find the blog post or CHANGES, etc.
> >
> > Kenn
> >
> > On Mon, Sep 27, 2021 at 2:16 PM Udi Meiri  wrote:
> >>
> >> I don't know how rare it is, but there is a flag documented in CHANGES
> and blog post that reverts to the old behavior.
> >>
> >> On Mon, Sep 27, 2021 at 2:12 PM Kenneth Knowles 
> wrote:
> >>>
> >>> I guess my vote is -0 since I don't have enough context on this issue.
> A number of people with more awareness of how severe this is have voted +1
> so I will not try to block the release.
> >>>
> >>> Kenn
> >>>
> >>> On Mon, Sep 27, 2021 at 2:11 PM Kenneth Knowles 
> wrote:
> 
>  I have to disagree with the other PMC members here, or at least dig
> in to the question: every pipeline that uses a Repeatedly trigger at the
> top level will be rejected. Is this so rare in Python that it is OK?
> 
>  Kenn
> 
>  On Mon, Sep 27, 2021 at 1:56 PM Robert Burke  wrote:
> >
> > On it. Thanks!
> >
> > On Mon, Sep 27, 2021 at 1:18 PM Udi Meiri  wrote:
> >>
> >> Daniel/Robert, feel free to make changes to this PR:
> https://github.com/apache/beam/pull/15543
> >>
> >> On Mon, Sep 27, 2021 at 12:08 PM Daniel Oliveira <
> danolive...@google.com> wrote:
> >>>
> >>> +1
> >>>
> >>> I hadn't realized the pipelines were still finishing successfully.
> I retried wordcount with that in mind and confirmed it finishes
> successfully, so this isn't a blocker.
> >>>
> >>> Although maybe we should add this to the "Known Issues" because I
> can easily see those messages being interpreted as a pipeline failure.
> >>>
> >>> On Sun, Sep 26, 2021 at 7:26 PM Robert Burke 
> wrote:
> 
>  That's on the SDK side, and it just means the PCollection metrics
> are being returned, buy not handled by the SDK. At present the pipeline
> results only handle PTransform metrics.
> 
>  As such it's not a regression, as adding those is still under
> development.
> 
>  On Sun, Sep 26, 2021, 7:02 PM Daniel Oliveira <
> danolive...@google.com> wrote:
> >
> > I tried validating wordcount with the Go SDK on Flink. The
> pipeline failed with a wall of errors like the following. I tried this on
> Flink 1.11, 1.12, and 1.13 job servers built from source at the RC1 commit,
> same errors on all of them.
> >
> >> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
> payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n9"}
> >> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
> payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n10"}
> >> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
> payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n8"}
> >> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
> payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n7"}
> >> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:sampled_byte_size:v1"
> type:"beam:metrics:distribution_int64:v1"
> payload:"\x8d%\xfa\xbf\x05\x04\xa8\x0c"  labels:{key:"PCOLLECTION"
> value:"n7"}
> >> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:sampled_byte_size:v1"
> type:"beam:metrics:distribution_int64:v1"  payload:"\xb9\x05\xcf6\x05\x11"
> labels:{key:"PCOLLECTION"  value:"n9"}
> >> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:sampled_byte_size:v1"
> type:"beam:metrics:distribution_int64:v1"  payload:"\xc5\x05\x8a1\x04\x12"
> labels:{key:"PCOLLECTION"  value:"n8"}
> >> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:sampled_byte_size:v1"
> type:"beam:metrics:distribution_int64:v1"
> payload:"\x8d%\xb3\xa7\x07\x14\""  labels:{key:"PCOLLECTION"  value:"n10"}
> >> {...}
> >
> >
> >  @Robert Burke I think you might know what's going on here. Is
> this solvable with a cherry-pick and a new RC?
> >
> > On Fri, Sep 24, 2021 at 4:25 PM Ahmet 

Re: [VOTE] Release 2.33.0, release candidate 1

[Kenneth Knowles]

My concern is that the error message is incorrect and every user of 2.33.0
may be educated wrong, or be worried about data loss in Beam, or fail to
find the blog post or CHANGES, etc.

Kenn

On Mon, Sep 27, 2021 at 2:16 PM Udi Meiri  wrote:

> I don't know how rare it is, but there is a flag documented in CHANGES and
> blog post that reverts to the old behavior.
>
> On Mon, Sep 27, 2021 at 2:12 PM Kenneth Knowles  wrote:
>
>> I guess my vote is -0 since I don't have enough context on this issue. A
>> number of people with more awareness of how severe this is have voted +1 so
>> I will not try to block the release.
>>
>> Kenn
>>
>> On Mon, Sep 27, 2021 at 2:11 PM Kenneth Knowles  wrote:
>>
>>> I have to disagree with the other PMC members here, or at least dig in
>>> to the question: every pipeline that uses a Repeatedly trigger at the top
>>> level will be rejected. Is this so rare in Python that it is OK?
>>>
>>> Kenn
>>>
>>> On Mon, Sep 27, 2021 at 1:56 PM Robert Burke  wrote:
>>>
 On it. Thanks!

 On Mon, Sep 27, 2021 at 1:18 PM Udi Meiri  wrote:

> Daniel/Robert, feel free to make changes to this PR:
> https://github.com/apache/beam/pull/15543
>
> On Mon, Sep 27, 2021 at 12:08 PM Daniel Oliveira <
> danolive...@google.com> wrote:
>
>> +1
>>
>> I hadn't realized the pipelines were still finishing successfully. I
>> retried wordcount with that in mind and confirmed it finishes 
>> successfully,
>> so this isn't a blocker.
>>
>> Although maybe we should add this to the "Known Issues" because I can
>> easily see those messages being interpreted as a pipeline failure.
>>
>> On Sun, Sep 26, 2021 at 7:26 PM Robert Burke 
>> wrote:
>>
>>> That's on the SDK side, and it just means the PCollection metrics
>>> are being returned, buy not handled by the SDK. At present the pipeline
>>> results only handle PTransform metrics.
>>>
>>> As such it's not a regression, as adding those is still under
>>> development.
>>>
>>> On Sun, Sep 26, 2021, 7:02 PM Daniel Oliveira <
>>> danolive...@google.com> wrote:
>>>
 I tried validating wordcount with the Go SDK on Flink. The pipeline
 failed with a wall of errors like the following. I tried this on Flink
 1.11, 1.12, and 1.13 job servers built from source at the RC1 commit, 
 same
 errors on all of them.

 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n9"}
> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n10"}
> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n8"}
> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n7"}
> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:sampled_byte_size:v1"
>  type:"beam:metrics:distribution_int64:v1"
>  payload:"\x8d%\xfa\xbf\x05\x04\xa8\x0c"  labels:{key:"PCOLLECTION"
>  value:"n7"}
> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:sampled_byte_size:v1"
>  type:"beam:metrics:distribution_int64:v1"  
> payload:"\xb9\x05\xcf6\x05\x11"
>  labels:{key:"PCOLLECTION"  value:"n9"}
> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:sampled_byte_size:v1"
>  type:"beam:metrics:distribution_int64:v1"  
> payload:"\xc5\x05\x8a1\x04\x12"
>  labels:{key:"PCOLLECTION"  value:"n8"}
> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
> urn:"beam:metric:sampled_byte_size:v1"
>  type:"beam:metrics:distribution_int64:v1"
>  payload:"\x8d%\xb3\xa7\x07\x14\""  labels:{key:"PCOLLECTION"  
> value:"n10"}
> {...}


  @Robert Burke  I think you might know what's
 going on here. Is this solvable with a cherry-pick and a new RC?

 On Fri, Sep 24, 2021 at 4:25 PM Ahmet Altay 
 wrote:

>
>
> On Fri, Sep 24, 2021 at 4:21 PM Udi Meiri 
> wrote:
>
>>
>>
>> On Fri, Sep 24, 2021 at 4:18 PM Ahmet Altay 
>> wrote:
>>
>>> Thank you for the update related to allow_unsafe_triggers. My
>>> vote is still a +1.
>>>

Re: [VOTE] Release 2.33.0, release candidate 1

[Kenneth Knowles]

I guess my vote is -0 since I don't have enough context on this issue. A
number of people with more awareness of how severe this is have voted +1 so
I will not try to block the release.

Kenn

On Mon, Sep 27, 2021 at 2:11 PM Kenneth Knowles  wrote:

> I have to disagree with the other PMC members here, or at least dig in to
> the question: every pipeline that uses a Repeatedly trigger at the top
> level will be rejected. Is this so rare in Python that it is OK?
>
> Kenn
>
> On Mon, Sep 27, 2021 at 1:56 PM Robert Burke  wrote:
>
>> On it. Thanks!
>>
>> On Mon, Sep 27, 2021 at 1:18 PM Udi Meiri  wrote:
>>
>>> Daniel/Robert, feel free to make changes to this PR:
>>> https://github.com/apache/beam/pull/15543
>>>
>>> On Mon, Sep 27, 2021 at 12:08 PM Daniel Oliveira 
>>> wrote:
>>>
 +1

 I hadn't realized the pipelines were still finishing successfully. I
 retried wordcount with that in mind and confirmed it finishes successfully,
 so this isn't a blocker.

 Although maybe we should add this to the "Known Issues" because I can
 easily see those messages being interpreted as a pipeline failure.

 On Sun, Sep 26, 2021 at 7:26 PM Robert Burke 
 wrote:

> That's on the SDK side, and it just means the PCollection metrics are
> being returned, buy not handled by the SDK. At present the pipeline 
> results
> only handle PTransform metrics.
>
> As such it's not a regression, as adding those is still under
> development.
>
> On Sun, Sep 26, 2021, 7:02 PM Daniel Oliveira 
> wrote:
>
>> I tried validating wordcount with the Go SDK on Flink. The pipeline
>> failed with a wall of errors like the following. I tried this on Flink
>> 1.11, 1.12, and 1.13 job servers built from source at the RC1 commit, 
>> same
>> errors on all of them.
>>
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>>> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>>>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n9"}
>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>>> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>>>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n10"}
>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>>> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>>>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n8"}
>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>>> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>>>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n7"}
>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>>> urn:"beam:metric:sampled_byte_size:v1"
>>>  type:"beam:metrics:distribution_int64:v1"
>>>  payload:"\x8d%\xfa\xbf\x05\x04\xa8\x0c"  labels:{key:"PCOLLECTION"
>>>  value:"n7"}
>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>>> urn:"beam:metric:sampled_byte_size:v1"
>>>  type:"beam:metrics:distribution_int64:v1"  
>>> payload:"\xb9\x05\xcf6\x05\x11"
>>>  labels:{key:"PCOLLECTION"  value:"n9"}
>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>>> urn:"beam:metric:sampled_byte_size:v1"
>>>  type:"beam:metrics:distribution_int64:v1"  
>>> payload:"\xc5\x05\x8a1\x04\x12"
>>>  labels:{key:"PCOLLECTION"  value:"n8"}
>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>>> urn:"beam:metric:sampled_byte_size:v1"
>>>  type:"beam:metrics:distribution_int64:v1"
>>>  payload:"\x8d%\xb3\xa7\x07\x14\""  labels:{key:"PCOLLECTION"  
>>> value:"n10"}
>>> {...}
>>
>>
>>  @Robert Burke  I think you might know what's going
>> on here. Is this solvable with a cherry-pick and a new RC?
>>
>> On Fri, Sep 24, 2021 at 4:25 PM Ahmet Altay  wrote:
>>
>>>
>>>
>>> On Fri, Sep 24, 2021 at 4:21 PM Udi Meiri  wrote:
>>>


 On Fri, Sep 24, 2021 at 4:18 PM Ahmet Altay 
 wrote:

> Thank you for the update related to allow_unsafe_triggers. My vote
> is still a +1.
>
> If we decide to move forward with this RC, could you please
> include this bug in the known issues list under the changes.md for 
> this
> release?
>
 Yes, that's included in
 https://github.com/apache/beam/pull/15543/files.

>>>
>>> Thank you!
>>>
>>>


>
> On Fri, Sep 24, 2021 at 3:56 PM Chamikara Jayalath <
> chamik...@google.com> wrote:
>
>> +1 (binding)
>> Validated a few scenarios from the spreadsheet.
>>
>> Thanks,
>> Cham
>>
>> On Fri, Sep 24, 2021 at 3:07 PM Robert Bradshaw <
>> rober...@google.com> wrote:

Re: [VOTE] Release 2.33.0, release candidate 1

[Kenneth Knowles]

I have to disagree with the other PMC members here, or at least dig in to
the question: every pipeline that uses a Repeatedly trigger at the top
level will be rejected. Is this so rare in Python that it is OK?

Kenn

On Mon, Sep 27, 2021 at 1:56 PM Robert Burke  wrote:

> On it. Thanks!
>
> On Mon, Sep 27, 2021 at 1:18 PM Udi Meiri  wrote:
>
>> Daniel/Robert, feel free to make changes to this PR:
>> https://github.com/apache/beam/pull/15543
>>
>> On Mon, Sep 27, 2021 at 12:08 PM Daniel Oliveira 
>> wrote:
>>
>>> +1
>>>
>>> I hadn't realized the pipelines were still finishing successfully. I
>>> retried wordcount with that in mind and confirmed it finishes successfully,
>>> so this isn't a blocker.
>>>
>>> Although maybe we should add this to the "Known Issues" because I can
>>> easily see those messages being interpreted as a pipeline failure.
>>>
>>> On Sun, Sep 26, 2021 at 7:26 PM Robert Burke  wrote:
>>>
 That's on the SDK side, and it just means the PCollection metrics are
 being returned, buy not handled by the SDK. At present the pipeline results
 only handle PTransform metrics.

 As such it's not a regression, as adding those is still under
 development.

 On Sun, Sep 26, 2021, 7:02 PM Daniel Oliveira 
 wrote:

> I tried validating wordcount with the Go SDK on Flink. The pipeline
> failed with a wall of errors like the following. I tried this on Flink
> 1.11, 1.12, and 1.13 job servers built from source at the RC1 commit, same
> errors on all of them.
>
> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n9"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n10"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n8"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:element_count:v1"  type:"beam:metrics:sum_int64:v1"
>>  payload:"\x8d%"  labels:{key:"PCOLLECTION"  value:"n7"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:sampled_byte_size:v1"
>>  type:"beam:metrics:distribution_int64:v1"
>>  payload:"\x8d%\xfa\xbf\x05\x04\xa8\x0c"  labels:{key:"PCOLLECTION"
>>  value:"n7"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:sampled_byte_size:v1"
>>  type:"beam:metrics:distribution_int64:v1"  
>> payload:"\xb9\x05\xcf6\x05\x11"
>>  labels:{key:"PCOLLECTION"  value:"n9"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:sampled_byte_size:v1"
>>  type:"beam:metrics:distribution_int64:v1"  
>> payload:"\xc5\x05\x8a1\x04\x12"
>>  labels:{key:"PCOLLECTION"  value:"n8"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:sampled_byte_size:v1"
>>  type:"beam:metrics:distribution_int64:v1"
>>  payload:"\x8d%\xb3\xa7\x07\x14\""  labels:{key:"PCOLLECTION"  
>> value:"n10"}
>> {...}
>
>
>  @Robert Burke  I think you might know what's going
> on here. Is this solvable with a cherry-pick and a new RC?
>
> On Fri, Sep 24, 2021 at 4:25 PM Ahmet Altay  wrote:
>
>>
>>
>> On Fri, Sep 24, 2021 at 4:21 PM Udi Meiri  wrote:
>>
>>>
>>>
>>> On Fri, Sep 24, 2021 at 4:18 PM Ahmet Altay 
>>> wrote:
>>>
 Thank you for the update related to allow_unsafe_triggers. My vote
 is still a +1.

 If we decide to move forward with this RC, could you please include
 this bug in the known issues list under the changes.md for this 
 release?

>>> Yes, that's included in
>>> https://github.com/apache/beam/pull/15543/files.
>>>
>>
>> Thank you!
>>
>>
>>>
>>>

 On Fri, Sep 24, 2021 at 3:56 PM Chamikara Jayalath <
 chamik...@google.com> wrote:

> +1 (binding)
> Validated a few scenarios from the spreadsheet.
>
> Thanks,
> Cham
>
> On Fri, Sep 24, 2021 at 3:07 PM Robert Bradshaw <
> rober...@google.com> wrote:
>
>> +1 (binding)
>>
>> All the artifacts and signatures look good.
>>
>> I don't think the unsafe trigger check is severe enough to block
>> the release.
>>
>> On Fri, Sep 24, 2021 at 2:36 PM Udi Meiri 
>> wrote:
>> >
>> > Hi everyone, we found a bug during testing. It has to do with
>> Python SDK's 

[RESULT] [VOTE] Release 2.33.0, release candidate #1

[Udi Meiri]

I'm happy to announce that we have unanimously approved this release.

There are 8 approving votes, 4 of which are binding:
* Ahmet Altay
* Alexey Romanenko
* Robert Bradshaw
* Chamikara Jayalath

There are no disapproving votes.

Thanks everyone!

Flaky test issue report (30)

[Beam Jira Bot]

This is your daily summary of Beam's current flaky tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake)

These are P1 issues because they have a major negative impact on the community 
and make it hard to determine the quality of the software.

https://issues.apache.org/jira/browse/BEAM-12928: beam_PostCommit_Python36 
- CrossLanguageSpannerIOTest - flakey failing (created 2021-09-21)
https://issues.apache.org/jira/browse/BEAM-12912: 
:runners:direct-java:runMobileGamingJavaDirect flakey failure in 
PostRelease_NightlySnapshot (created 2021-09-17)
https://issues.apache.org/jira/browse/BEAM-12861: 
apache_beam.ml.gcp.recommendations_ai_test_it.RecommendationAIIT.test_create_catalog_item
  is flaky (created 2021-09-09)
https://issues.apache.org/jira/browse/BEAM-12859: 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
 is flaky (created 2021-09-08)
https://issues.apache.org/jira/browse/BEAM-12809: 
testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky (created 2021-08-26)
https://issues.apache.org/jira/browse/BEAM-12694: DICOMIoIntegrationTest 
flaky due to store ID (Python PreCommit) (created 2021-07-30)
https://issues.apache.org/jira/browse/BEAM-12515: Python PreCommit flaking 
in PipelineOptionsTest.test_display_data (created 2021-06-18)
https://issues.apache.org/jira/browse/BEAM-12322: Python precommit flaky: 
Failed to read inputs in the data plane (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12320: 
PubsubTableProviderIT.testSQLSelectsArrayAttributes[0] failing in SQL 
PostCommit (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12291: 
org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: 
false] is flaky (created 2021-05-05)
https://issues.apache.org/jira/browse/BEAM-12200: 
SamzaStoreStateInternalsTest is flaky (created 2021-04-20)
https://issues.apache.org/jira/browse/BEAM-12163: Python GHA PreCommits 
flake with grpc.FutureTimeoutError on SDK harness startup (created 2021-04-13)
https://issues.apache.org/jira/browse/BEAM-12061: beam_PostCommit_SQL 
failing on KafkaTableProviderIT.testFakeNested (created 2021-03-27)
https://issues.apache.org/jira/browse/BEAM-11837: Java build flakes: 
"Memory constraints are impeding performance" (created 2021-02-18)
https://issues.apache.org/jira/browse/BEAM-11661: hdfsIntegrationTest 
flake: network not found (py38 postcommit) (created 2021-01-19)
https://issues.apache.org/jira/browse/BEAM-11645: beam_PostCommit_XVR_Flink 
failing (created 2021-01-15)
https://issues.apache.org/jira/browse/BEAM-11641: Bigquery Read tests are 
flaky on Flink runner in Python PostCommit suites (created 2021-01-15)
https://issues.apache.org/jira/browse/BEAM-11541: 
testTeardownCalledAfterExceptionInProcessElement flakes on direct runner. 
(created 2020-12-30)
https://issues.apache.org/jira/browse/BEAM-10955: Flink Java Runner test 
flake: Could not find Flink job (FlinkJobNotFoundException) (created 2020-09-23)
https://issues.apache.org/jira/browse/BEAM-10866: 
PortableRunnerTestWithSubprocesses.test_register_finalizations flaky on macOS 
(created 2020-09-09)
https://issues.apache.org/jira/browse/BEAM-10485: Failure / flake: 
ElasticsearchIOTest > testWriteWithIndexFn (created 2020-07-14)
https://issues.apache.org/jira/browse/BEAM-9649: 
beam_python_mongoio_load_test started failing due to mismatched results 
(created 2020-03-31)
https://issues.apache.org/jira/browse/BEAM-8453: Failure in 
org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMarkSafety (created 
2019-10-21)
https://issues.apache.org/jira/browse/BEAM-8101: Flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful for 
Direct, Spark, Flink (created 2019-08-27)
https://issues.apache.org/jira/browse/BEAM-8035: 
WatchTest.testMultiplePollsWithManyResults flake: Outputs must be in timestamp 
order (sickbayed) (created 2019-08-22)
https://issues.apache.org/jira/browse/BEAM-7827: 
MetricsTest$AttemptedMetricTests.testAllAttemptedMetrics is flaky on 
DirectRunner (created 2019-07-26)
https://issues.apache.org/jira/browse/BEAM-7752: Java Validates 
DirectRunner: testTeardownCalledAfterExceptionInFinishBundleStateful flaky 
(created 2019-07-16)
https://issues.apache.org/jira/browse/BEAM-6804: [beam_PostCommit_Java] 
[PubsubReadIT.testReadPublicData] Timeout waiting on Sub (created 2019-03-11)
https://issues.apache.org/jira/browse/BEAM-5286: 
[beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake]
 .sh script: text file busy. (created 2018-09-01)
https://issues.apache.org/jira/browse/BEAM-5172: 
org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky (created 
2018-08-20)

Re: Create a Dataset in GCP Testing Project?

[Robert Burke]

We should probably add something to the wiki for that.

On Mon, Sep 27, 2021, 10:42 AM Brian Hulette  wrote:

> I don't think there's any policy in place for controlling access to the
> apache-beam-testing project. I think in general PMC members are owners and
> committers are editors, but it looks like there are a lot of exceptions to
> this rule. For example, I am an owner - so I was able to grant you editor
> access. I think you should be able to create a new dataset now.
>
> Brian
>
> On Fri, Sep 24, 2021 at 12:07 PM Austin Bennett <
> whatwouldausti...@gmail.com> wrote:
>
>> Hi Devs,
>>
>> I am working on https://issues.apache.org/jira/browse/BEAM-10652 and
>> specifically sorting out Integration Tests for this.  I believe that I need
>> to create a dataset for this to work given errors ( a special purpose
>> dataset seems cleaner than reusing an existing dataset, and in line with
>> the conventions I see ).
>>
>> I would like to work through this, rather than someone just handling it.
>>
>> How to get access to our GCP Projects used for testing?  I might also
>> have questions for how we generally like things done within ( ex: I haven't
>> seen terraform repose for how we manage that infrastructure ;-) ).
>>
>> Thanks,
>> Austin
>>
>>

P1 issues report (46)

[Beam Jira Bot]

This is your daily summary of Beam's current P1 issues, not including flaky 
tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake).

See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the 
meaning and expectations around P1 issues.

https://issues.apache.org/jira/browse/BEAM-12950: Missing events when using 
Python WriteToFiles in streaming pipeline (created 2021-09-24)
https://issues.apache.org/jira/browse/BEAM-12867: Either Create or 
DirectRunner fails to produce all elements to the following transform (created 
2021-09-09)
https://issues.apache.org/jira/browse/BEAM-12843: (Broken Pipe induced) 
Bricked Dataflow Pipeline  (created 2021-09-06)
https://issues.apache.org/jira/browse/BEAM-12807: Java creates an incorrect 
pipeline proto when core-construction-java jar is not in the CLASSPATH (created 
2021-08-26)
https://issues.apache.org/jira/browse/BEAM-12794: 
PortableRunnerTestWithExternalEnv.test_pardo_timers flaky (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12792: Beam worker only installs 
--extra_package once (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12781: SDFBoundedSourceReader 
behaves much slower compared with the original behavior of BoundedSource 
(created 2021-08-20)
https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset 
apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16)
https://issues.apache.org/jira/browse/BEAM-12632: ElasticsearchIO: Enabling 
both User/Pass auth and SSL overwrites User/Pass (created 2021-07-16)
https://issues.apache.org/jira/browse/BEAM-12628: AvroCoder changed 
underlying String class for SpecificRecords (created 2021-07-16)
https://issues.apache.org/jira/browse/BEAM-12607: Copy Code Snippet copies 
html tags (created 2021-07-13)
https://issues.apache.org/jira/browse/BEAM-12540: 
beam_PostRelease_NightlySnapshot - Task 
:runners:direct-java:runMobileGamingJavaDirect FAILED (created 2021-06-25)
https://issues.apache.org/jira/browse/BEAM-12525: SDF BoundedSource seems 
to execute significantly slower than 'normal' BoundedSource (created 2021-06-22)
https://issues.apache.org/jira/browse/BEAM-12505: codecov/patch has poor 
behavior (created 2021-06-17)
https://issues.apache.org/jira/browse/BEAM-12500: Dataflow SocketException 
(SSLException) error while trying to send message from Cloud Pub/Sub to 
BigQuery (created 2021-06-16)
https://issues.apache.org/jira/browse/BEAM-12484: JdbcIO date conversion is 
sensitive to OS (created 2021-06-14)
https://issues.apache.org/jira/browse/BEAM-12467: 
java.io.InvalidClassException With Flink Kafka (created 2021-06-09)
https://issues.apache.org/jira/browse/BEAM-12380: Go SDK Kafka IO Transform 
implemented via XLang (created 2021-05-21)
https://issues.apache.org/jira/browse/BEAM-12310: 
beam_PostCommit_Java_DataflowV2 failing (created 2021-05-07)
https://issues.apache.org/jira/browse/BEAM-12279: Implement 
destination-dependent sharding in FileIO.writeDynamic (created 2021-05-04)
https://issues.apache.org/jira/browse/BEAM-12256: 
PubsubIO.readAvroGenericRecord creates SchemaCoder that fails to decode some 
Avro logical types (created 2021-04-29)
https://issues.apache.org/jira/browse/BEAM-11959: Python Beam SDK Harness 
hangs when installing pip packages (created 2021-03-11)
https://issues.apache.org/jira/browse/BEAM-11906: No trigger early 
repeatedly for session windows (created 2021-03-01)
https://issues.apache.org/jira/browse/BEAM-11875: XmlIO.Read does not 
handle XML encoding per spec (created 2021-02-26)
https://issues.apache.org/jira/browse/BEAM-11828: JmsIO is not 
acknowledging messages correctly (created 2021-02-17)
https://issues.apache.org/jira/browse/BEAM-11755: Cross-language 
consistency (RequiresStableInputs) is quietly broken (at least on portable 
flink runner) (created 2021-02-05)
https://issues.apache.org/jira/browse/BEAM-11578: `dataflow_metrics` 
(python) fails with TypeError (when int overflowing?) (created 2021-01-06)
https://issues.apache.org/jira/browse/BEAM-11148: Kafka 
commitOffsetsInFinalize OOM on Flink (created 2020-10-28)
https://issues.apache.org/jira/browse/BEAM-11017: Timer with dataflow 
runner can be set multiple times (dataflow runner) (created 2020-10-05)
https://issues.apache.org/jira/browse/BEAM-10670: Make non-portable 
Splittable DoFn the only option when executing Java "Read" transforms (created 
2020-08-10)
https://issues.apache.org/jira/browse/BEAM-10617: python 
CombineGlobally().with_fanout() cause duplicate combine results for sliding 
windows (created 2020-07-31)
https://issues.apache.org/jira/browse/BEAM-10569: SpannerIO tests don't 
actually assert anything. (created 2020-07-23)

P0 (outage) report

[Beam Jira Bot]

This is your daily summary of Beam's current outages. See 
https://beam.apache.org/contribute/jira-priorities/#p0-outage for the meaning 
and expectations around P0 issues.

BEAM-12959: Dataflow error in CombinePerKey operation 
(https://issues.apache.org/jira/browse/BEAM-12959)

Beam Dependency Check Report (2021-09-27)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
chromedriver-binary
93.0.4577.63.0
95.0.4638.10.0
2021-09-20
2021-09-27BEAM-10426
dill
0.3.1.1
0.3.4
2019-10-07
2021-06-14BEAM-11167
google-api-core
1.31.3
2.0.1
None
2021-09-06BEAM-12784
google-auth
1.35.0
2.1.0
2021-08-23
2021-09-20BEAM-12785
google-cloud-bigtable
1.7.0
2.3.3
2021-04-12
2021-08-02BEAM-8127
google-cloud-datastore
1.15.3
2.1.6
2020-11-16
2021-08-02BEAM-8443
google-cloud-dlp
1.0.0
3.2.2
2020-06-29
2021-08-02BEAM-10344
google-cloud-language
1.3.0
2.2.2
2020-10-26
2021-08-02BEAM-8
google-cloud-pubsub
1.7.0
2.8.0
2020-07-20
2021-09-06BEAM-5539
google-cloud-spanner
1.19.1
3.10.0
2020-11-16
2021-09-27BEAM-10345
google-cloud-videointelligence
1.16.1
2.3.2
2020-11-23
2021-08-02BEAM-11319
google-cloud-vision
1.0.0
2.4.2
2020-03-24
2021-08-02BEAM-9581
ipykernel
5.5.5
6.4.1
2021-07-05
2021-09-13BEAM-12575
jupyter-client
6.1.12
7.0.3
2021-04-12
2021-09-20BEAM-12786
mock
2.0.0
4.0.3
2019-05-20
2020-12-14BEAM-7369
mypy-protobuf
1.18
2.10
2020-03-24
2021-09-20BEAM-10346
nbconvert
5.6.1
6.2.0
2020-10-05
2021-09-27BEAM-11007
Pillow
7.2.0
8.3.2
2020-10-19
2021-09-06BEAM-11071
pluggy
0.13.1
1.0.0
2021-08-30
2021-08-30BEAM-12819
PyHamcrest
1.10.1
2.0.2
2020-01-20
2020-07-08BEAM-9155
pytest
4.6.11
6.2.5
2020-07-08
2021-09-06BEAM-8606
pytest-xdist
1.34.0
2.4.0
2020-08-17
2021-09-27BEAM-10713
setuptools
57.4.0
58.1.0
2021-07-19
2021-09-27BEAM-10714
tenacity
5.1.5
8.0.1
2019-11-11
2021-07-19BEAM-8607
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.alibaba:fastjson
1.2.69
1.2.78
2020-05-31
2021-08-09BEAM-8632
com.azure:azure-core
1.9.0
1.21.0-beta.1
2020-10-02
2021-09-08BEAM-11888
com.azure:azure-identity
1.0.8
1.4.0-beta.1
2020-07-07
2021-09-13BEAM-11814
com.azure:azure-storage-common
12.10.0
12.13.0
2021-01-14
2021-09-15BEAM-11889
com.datastax.cassandra:cassandra-driver-core
3.10.2
4.0.0
2020-08-26
2019-03-18BEAM-8674
com.esotericsoftware:kryo
4.0.2
5.2.0
2018-03-20
2021-07-31BEAM-5809
com.esotericsoftware.kryo:kryo
2.21
2.24.0
2013-02-27
2014-05-04BEAM-5574
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.33.0
0.39.0
2020-09-14
2021-05-31BEAM-6645
com.github.jk1.dependency-license-report:com.github.jk1.dependency-license-report.gradle.plugin
1.16
2.0
2020-10-26
2021-09-06BEAM-11120
com.github.luben:zstd-jni
1.4.5-2
1.5.0-4
2020-05-30
2021-07-21BEAM-9194
com.github.spotbugs:spotbugs
4.0.6
4.4.1
2020-06-23
2021-09-10BEAM-7792
com.github.spotbugs:spotbugs-annotations
4.0.6
4.4.1
2020-06-23
2021-09-10BEAM-6951
com.google.api.grpc:proto-google-cloud-dlp-v2
1.1.4
3.0.7
2020-05-04
2021-09-22BEAM-11892
com.google.api.grpc:proto-google-cloud-recommendations-ai-v1beta1
0.3.7
0.8.1
2020-11-02
2021-09-22BEAM-12517
com.google.api.grpc:proto-google-cloud-video-intelligence-v1
1.2.0
2.0.7
2020-03-10
2021-09-20BEAM-11894
com.google.api.grpc:proto-google-cloud-vision-v1
1.81.3
2.0.13
2020-04-07
2021-09-24BEAM-11895
com.google.apis:google-api-services-bigquery
v2-rev20210813-1.32.1