Re: Go error when building containers

[Daniel Oliveira]

It looks like the cached version of a package is stale and causing build
errors when building beam. Chances are just deleting that
/.gogradle directory will cause everything to rebuild from a clean state,
so I'd try that. I think it should be /sdks/go/.gogradle

On Thu, Jul 23, 2020 at 4:56 PM Ahmet Altay  wrote:

> This is probably : https://issues.apache.org/jira/browse/BEAM-10567
> 
>
> On Thu, Jul 23, 2020 at 4:53 PM Brian Hulette  wrote:
>
>> Whenever I build a container locally
>> (:sdks:java:container:docker, :sdks:python:container:py37:docker, ..) I get
>> a Go error (log at the end of this message).
>>
>> I've discovered I can just comment out resolveBuildDependencies.dependsOn
>> ":sdks:go:goBuild" in the relevant build.gradle file [1] whenever this
>> happens, but it's getting old and I'm wondering if there's a better way. Is
>> there something wrong with my environment that's causing these errors (It
>> must not be an actual breakage in the Go SDK)? Can we remove or modify this
>> statement to fix this?
>>
>> Thanks,
>> Brian
>>
>> [1]
>> https://github.com/apache/beam/blob/59b7200c8621b81804d53ded771fd3aa525fbb47/sdks/java/container/build.gradle#L32
>>
>> # github.com/apache/beam/sdks/go/test/integration/synthetic
>> .gogradle/project_gopath/src/
>> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:31:31:
>> cannot use s (type "
>> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
>> as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
>> synthetic.SourceSingle
>> .gogradle/project_gopath/src/
>> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:33:24:
>> cannot use s (type "
>> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
>> as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
>> synthetic.Step
>> .gogradle/project_gopath/src/
>> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:34:2:
>> undefined: passert.Count
>> .gogradle/project_gopath/src/
>> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:51:25:
>> cannot use s (type "
>> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
>> as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
>> synthetic.Source
>> .gogradle/project_gopath/src/
>> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:51:25:
>> cannot use configs (type "
>> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".PCollection)
>> as type "github.com/apache/beam/sdks/go/pkg/beam".PCollection in
>> argument to synthetic.Source
>> .gogradle/project_gopath/src/
>> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:52:24:
>> cannot use s (type "
>> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
>> as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
>> synthetic.Step
>> .gogradle/project_gopath/src/
>> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:61:2:
>> undefined: passert.Count
>>
>> > Task :sdks:go:buildLinuxAmd64 FAILED
>>
>

Re: Go error when building containers

[Ahmet Altay]

This is probably : https://issues.apache.org/jira/browse/BEAM-10567


On Thu, Jul 23, 2020 at 4:53 PM Brian Hulette  wrote:

> Whenever I build a container locally
> (:sdks:java:container:docker, :sdks:python:container:py37:docker, ..) I get
> a Go error (log at the end of this message).
>
> I've discovered I can just comment out resolveBuildDependencies.dependsOn
> ":sdks:go:goBuild" in the relevant build.gradle file [1] whenever this
> happens, but it's getting old and I'm wondering if there's a better way. Is
> there something wrong with my environment that's causing these errors (It
> must not be an actual breakage in the Go SDK)? Can we remove or modify this
> statement to fix this?
>
> Thanks,
> Brian
>
> [1]
> https://github.com/apache/beam/blob/59b7200c8621b81804d53ded771fd3aa525fbb47/sdks/java/container/build.gradle#L32
>
> # github.com/apache/beam/sdks/go/test/integration/synthetic
> .gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:31:31:
> cannot use s (type "
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
> as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
> synthetic.SourceSingle
> .gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:33:24:
> cannot use s (type "
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
> as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
> synthetic.Step
> .gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:34:2:
> undefined: passert.Count
> .gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:51:25:
> cannot use s (type "
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
> as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
> synthetic.Source
> .gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:51:25:
> cannot use configs (type "
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".PCollection)
> as type "github.com/apache/beam/sdks/go/pkg/beam".PCollection in argument
> to synthetic.Source
> .gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:52:24:
> cannot use s (type "
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
> as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
> synthetic.Step
> .gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:61:2:
> undefined: passert.Count
>
> > Task :sdks:go:buildLinuxAmd64 FAILED
>

Go error when building containers

[Brian Hulette]

Whenever I build a container locally
(:sdks:java:container:docker, :sdks:python:container:py37:docker, ..) I get
a Go error (log at the end of this message).

I've discovered I can just comment out resolveBuildDependencies.dependsOn
":sdks:go:goBuild" in the relevant build.gradle file [1] whenever this
happens, but it's getting old and I'm wondering if there's a better way. Is
there something wrong with my environment that's causing these errors (It
must not be an actual breakage in the Go SDK)? Can we remove or modify this
statement to fix this?

Thanks,
Brian

[1]
https://github.com/apache/beam/blob/59b7200c8621b81804d53ded771fd3aa525fbb47/sdks/java/container/build.gradle#L32

# github.com/apache/beam/sdks/go/test/integration/synthetic
.gogradle/project_gopath/src/
github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:31:31:
cannot use s (type "
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
synthetic.SourceSingle
.gogradle/project_gopath/src/
github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:33:24:
cannot use s (type "
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
synthetic.Step
.gogradle/project_gopath/src/
github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:34:2:
undefined: passert.Count
.gogradle/project_gopath/src/
github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:51:25:
cannot use s (type "
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
synthetic.Source
.gogradle/project_gopath/src/
github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:51:25:
cannot use configs (type "
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".PCollection)
as type "github.com/apache/beam/sdks/go/pkg/beam".PCollection in argument
to synthetic.Source
.gogradle/project_gopath/src/
github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:52:24:
cannot use s (type "
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam".Scope)
as type "github.com/apache/beam/sdks/go/pkg/beam".Scope in argument to
synthetic.Step
.gogradle/project_gopath/src/
github.com/apache/beam/sdks/go/test/integration/synthetic/synthetic.go:61:2:
undefined: passert.Count

> Task :sdks:go:buildLinuxAmd64 FAILED

Re: [BROKEN] Please add "Fix Version" when resolving or closing Jiras

[Brian Hulette]

+1 - I think I saw somewhere Resolved is intended for "this is fixed" while
Closed is "the reporter/some authority has verified it's fixed". Doesn't
seem to apply to us.

On Thu, Jul 23, 2020 at 2:21 PM Kyle Weaver  wrote:

> > we could simplify those to a single state and let the Resolution field
> be the one source of truth for the nature of the resolution.
>
> +1
>
> On Thu, Jul 23, 2020 at 2:14 PM Kenneth Knowles  wrote:
>
>> This is unfortunate :-(
>>
>> Also unfortunate: I have just enough permissions to mess it up and not
>> enough to fix it. You can follow
>> https://issues.apache.org/jira/browse/INFRA-20563
>>
>> By the way, does anyone really use the difference between "Resolved" and
>> "Closed"? Since I am talking to Infra and probably getting our workflow
>> re-initialized manually, we could simplify those to a single state and let
>> the Resolution field be the one source of truth for the nature of the
>> resolution. (typically I think fixed things get "resolved" while wontfix
>> issues get "closed" - this is not actually what the difference is supposed
>> to be)
>>
>> Kenn
>>
>> On Thu, Jul 23, 2020 at 9:43 AM Brian Hulette 
>> wrote:
>>
>>> Yeah same for me. Not a huge deal.. I just noticed that the jira UI
>>> isn't rendering references to my closed tickets with a strikethrough, and
>>> I'd like to distinguish others as "won't fix" or "obsolete". It should be
>>> easy enough to backfill these once this is sorted out though.
>>>
>>> Brian
>>>
>>> On Thu, Jul 23, 2020 at 2:01 AM Maximilian Michels 
>>> wrote:
>>>
 I can close/resolve but it will show "Resolution: Unresolved":

 On 23.07.20 02:30, Brian Hulette wrote:

 Is setting the Resolution broken as well? I realized I've been closing
 jiras with Resolution "Unresolved" and I can't actually change it to
 "Fixed".

 On Tue, Jul 21, 2020 at 7:19 AM Maximilian Michels 
 wrote:

> Also, a friendly reminder to always close the JIRA issue after merging
> a
> fix. It's easy to forget.
>
> On 20.07.20 21:04, Kenneth Knowles wrote:
> > Hi all,
> >
> > In working on our Jira automation, I've messed up our Jira workflow.
> It
> > will no longer prompt you to fill in "Fix Version" when you resolve
> or
> > close an issue. I will be working with infra to restore this. In the
> > meantime, please try to remember to add a Fix Version to each issue
> that
> > you close, so that we get automated detailed release notes.
> >
> > Kenn
>

Re: [BROKEN] Please add "Fix Version" when resolving or closing Jiras

[Kyle Weaver]

> we could simplify those to a single state and let the Resolution field be
the one source of truth for the nature of the resolution.

+1

On Thu, Jul 23, 2020 at 2:14 PM Kenneth Knowles  wrote:

> This is unfortunate :-(
>
> Also unfortunate: I have just enough permissions to mess it up and not
> enough to fix it. You can follow
> https://issues.apache.org/jira/browse/INFRA-20563
>
> By the way, does anyone really use the difference between "Resolved" and
> "Closed"? Since I am talking to Infra and probably getting our workflow
> re-initialized manually, we could simplify those to a single state and let
> the Resolution field be the one source of truth for the nature of the
> resolution. (typically I think fixed things get "resolved" while wontfix
> issues get "closed" - this is not actually what the difference is supposed
> to be)
>
> Kenn
>
> On Thu, Jul 23, 2020 at 9:43 AM Brian Hulette  wrote:
>
>> Yeah same for me. Not a huge deal.. I just noticed that the jira UI isn't
>> rendering references to my closed tickets with a strikethrough, and I'd
>> like to distinguish others as "won't fix" or "obsolete". It should be easy
>> enough to backfill these once this is sorted out though.
>>
>> Brian
>>
>> On Thu, Jul 23, 2020 at 2:01 AM Maximilian Michels 
>> wrote:
>>
>>> I can close/resolve but it will show "Resolution: Unresolved":
>>>
>>> On 23.07.20 02:30, Brian Hulette wrote:
>>>
>>> Is setting the Resolution broken as well? I realized I've been closing
>>> jiras with Resolution "Unresolved" and I can't actually change it to
>>> "Fixed".
>>>
>>> On Tue, Jul 21, 2020 at 7:19 AM Maximilian Michels 
>>> wrote:
>>>
 Also, a friendly reminder to always close the JIRA issue after merging
 a
 fix. It's easy to forget.

 On 20.07.20 21:04, Kenneth Knowles wrote:
 > Hi all,
 >
 > In working on our Jira automation, I've messed up our Jira workflow.
 It
 > will no longer prompt you to fill in "Fix Version" when you resolve
 or
 > close an issue. I will be working with infra to restore this. In the
 > meantime, please try to remember to add a Fix Version to each issue
 that
 > you close, so that we get automated detailed release notes.
 >
 > Kenn

>>>

Re: Jenkins trigger phrase "run seed job" not working?

[Damian Gadomski]

Oh, with our new Jenkins that's not an issue - I have admin access there,
the issue is with checking old CI configuration, GitHub, infra stuff, etc.
FYI, all PMC members as well have admin access to the new CI and can
install plugins.

This 'fetching&whitelisting committers during seed job' solution should not
disable phrases. That will work as expected.

On Thu, Jul 23, 2020 at 10:58 PM Udi Meiri  wrote:

> I have the same issue with Jenkins privileges. There's usually no insight
> to test triggering logic.
> For instance I happen to know that tests won't be started right now
> because Infra is restarting Jenkins to install a plugin, but that's only
> because I opened the ticket.
>
> I think fetching the list of allowed user IDs as part of the seed job is
> okay. Even if this disables phrases we can always manually trigger the seed
> job from the Jenkins UI.
>
> On Thu, Jul 23, 2020 at 1:11 PM Damian Gadomski <
> damian.gadom...@polidea.com> wrote:
>
>> Yes, I thought that whitelisting apache organization will do the trick,
>> but apparently, it doesn't. Actually, it makes sense as we want to allow
>> only beam committers and not all apache committers. I don't know the
>> implications of membership in the apache github organization, but you for
>> instance are not there :) Neither is Ahmet.
>>
>>
>> Therefore there's nothing wrong with the Ghprb plugin, it correctly
>> forbade triggering. From my investigation, the "beam-committers" GitHub
>> team (which is under the apache org) is the list of people that should be
>> allowed. But firstly, you cant whitelist a team with Ghprb. There's a
>> ticket for that, open for 5 years
>> . I could
>> implement that but, secondly, the team is secret. I can't even see it. Even
>> asfbot doesn't have permission to see it.
>>
>> You may ask, how it worked before, because on the builds.apache.org
>> somehow only committers were allowed to trigger PR builds. It appeared that
>> Infra created a webhook relay. It's configured here
>> 
>>  and
>> it filters out all the non-committers events. I wish I had known that
>> before as it was also the reason for different issues during the migration.
>> Anyway, it would be hard to use that mechanism in our case as we want to
>> configure it depending on the job.
>>
>>
>> There's a publicly available source of committers list - it's LDAP. I've
>> tested it and it allows anonymous connection and provides the list of the
>> committers as well as the github usernames. My current idea is to read this
>> from LDAP as a part of the seed job and configure the jobs with the apache
>> committers present on the ghprb whitelist.
>>
>>
>> Hope that I didn't miss anything ;) It isn't that easy to investigate
>> that kind of issues with my poor privileges ;)
>>
>>
>> Regards,
>>
>> Damian
>>
>>
>> On Thu, Jul 23, 2020 at 6:52 PM Udi Meiri  wrote:
>>
>>> Thanks Damian! I saw that the config also has this:
>>>   orgWhitelist(['apache'])
>>> Shouldn't that be enough to allow all Apache committers?
>>>
>>> I traced the code for the membership check here:
>>>
>>> https://github.com/jenkinsci/ghprb-plugin/blob/4e86ed47a96a01eeaa51a479ff604252109635f6/src/main/java/org/jenkinsci/plugins/ghprb/GhprbGitHub.java#L27
>>> Is there a way to see these logs?
>>>
>>>
>>> On Thu, Jul 23, 2020 at 7:08 AM Damian Gadomski <
>>> damian.gadom...@polidea.com> wrote:
>>>
 Hi,

 You are right, the current behavior is wrong, I'm currently working to
 fix it asap. Our intention was to disable that only for non-committers.

 As a workaround, as a committer, you could manually add yourself (your
 GitHub username) to the whitelist of the SeedJob configuration:
 https://ci-beam.apache.org/job/beam_SeedJob/configure
 Then, your comment "Run Seed Job" will trigger the build. I've already
 manually triggered it for you that way.

 Of course, it will only work until the seed job gets executed - it will
 then override the whitelist with an empty one.

 [image: Selection_408.png]

 As a target solution, I'm planning to fetch the list of beam committers
 from LDAP and automatically add them to the whitelist above as a part of
 the seed job. I'll keep you updated about the progress.

 Regards,
 Damian


 On Wed, Jul 22, 2020 at 11:03 PM Ahmet Altay  wrote:

> +Damian Gadomski , it might be related
> to this change: https://github.com/apache/beam/pull/12319.
>
> /cc +Tyson Hamilton 
>
> On Wed, Jul 22, 2020 at 1:17 PM Udi Meiri  wrote:
>
>> HI,
>> I'm trying to test a groovy change but I can't seem to trigger the
>> seed job. It worked yesterday so I'm not sure what changed.
>>
>> https://github.com/apache/beam/pull/12326
>>
>>

Re: [BROKEN] Please add "Fix Version" when resolving or closing Jiras

[Kenneth Knowles]

This is unfortunate :-(

Also unfortunate: I have just enough permissions to mess it up and not
enough to fix it. You can follow
https://issues.apache.org/jira/browse/INFRA-20563

By the way, does anyone really use the difference between "Resolved" and
"Closed"? Since I am talking to Infra and probably getting our workflow
re-initialized manually, we could simplify those to a single state and let
the Resolution field be the one source of truth for the nature of the
resolution. (typically I think fixed things get "resolved" while wontfix
issues get "closed" - this is not actually what the difference is supposed
to be)

Kenn

On Thu, Jul 23, 2020 at 9:43 AM Brian Hulette  wrote:

> Yeah same for me. Not a huge deal.. I just noticed that the jira UI isn't
> rendering references to my closed tickets with a strikethrough, and I'd
> like to distinguish others as "won't fix" or "obsolete". It should be easy
> enough to backfill these once this is sorted out though.
>
> Brian
>
> On Thu, Jul 23, 2020 at 2:01 AM Maximilian Michels  wrote:
>
>> I can close/resolve but it will show "Resolution: Unresolved":
>>
>> On 23.07.20 02:30, Brian Hulette wrote:
>>
>> Is setting the Resolution broken as well? I realized I've been closing
>> jiras with Resolution "Unresolved" and I can't actually change it to
>> "Fixed".
>>
>> On Tue, Jul 21, 2020 at 7:19 AM Maximilian Michels 
>> wrote:
>>
>>> Also, a friendly reminder to always close the JIRA issue after merging a
>>> fix. It's easy to forget.
>>>
>>> On 20.07.20 21:04, Kenneth Knowles wrote:
>>> > Hi all,
>>> >
>>> > In working on our Jira automation, I've messed up our Jira workflow.
>>> It
>>> > will no longer prompt you to fill in "Fix Version" when you resolve or
>>> > close an issue. I will be working with infra to restore this. In the
>>> > meantime, please try to remember to add a Fix Version to each issue
>>> that
>>> > you close, so that we get automated detailed release notes.
>>> >
>>> > Kenn
>>>
>>

Re: Jenkins trigger phrase "run seed job" not working?

[Udi Meiri]

I have the same issue with Jenkins privileges. There's usually no insight
to test triggering logic.
For instance I happen to know that tests won't be started right now because
Infra is restarting Jenkins to install a plugin, but that's only because I
opened the ticket.

I think fetching the list of allowed user IDs as part of the seed job is
okay. Even if this disables phrases we can always manually trigger the seed
job from the Jenkins UI.

On Thu, Jul 23, 2020 at 1:11 PM Damian Gadomski 
wrote:

> Yes, I thought that whitelisting apache organization will do the trick,
> but apparently, it doesn't. Actually, it makes sense as we want to allow
> only beam committers and not all apache committers. I don't know the
> implications of membership in the apache github organization, but you for
> instance are not there :) Neither is Ahmet.
>
>
> Therefore there's nothing wrong with the Ghprb plugin, it correctly
> forbade triggering. From my investigation, the "beam-committers" GitHub
> team (which is under the apache org) is the list of people that should be
> allowed. But firstly, you cant whitelist a team with Ghprb. There's a
> ticket for that, open for 5 years
> . I could implement
> that but, secondly, the team is secret. I can't even see it. Even asfbot
> doesn't have permission to see it.
>
> You may ask, how it worked before, because on the builds.apache.org
> somehow only committers were allowed to trigger PR builds. It appeared that
> Infra created a webhook relay. It's configured here
> 
>  and
> it filters out all the non-committers events. I wish I had known that
> before as it was also the reason for different issues during the migration.
> Anyway, it would be hard to use that mechanism in our case as we want to
> configure it depending on the job.
>
>
> There's a publicly available source of committers list - it's LDAP. I've
> tested it and it allows anonymous connection and provides the list of the
> committers as well as the github usernames. My current idea is to read this
> from LDAP as a part of the seed job and configure the jobs with the apache
> committers present on the ghprb whitelist.
>
>
> Hope that I didn't miss anything ;) It isn't that easy to investigate that
> kind of issues with my poor privileges ;)
>
>
> Regards,
>
> Damian
>
>
> On Thu, Jul 23, 2020 at 6:52 PM Udi Meiri  wrote:
>
>> Thanks Damian! I saw that the config also has this:
>>   orgWhitelist(['apache'])
>> Shouldn't that be enough to allow all Apache committers?
>>
>> I traced the code for the membership check here:
>>
>> https://github.com/jenkinsci/ghprb-plugin/blob/4e86ed47a96a01eeaa51a479ff604252109635f6/src/main/java/org/jenkinsci/plugins/ghprb/GhprbGitHub.java#L27
>> Is there a way to see these logs?
>>
>>
>> On Thu, Jul 23, 2020 at 7:08 AM Damian Gadomski <
>> damian.gadom...@polidea.com> wrote:
>>
>>> Hi,
>>>
>>> You are right, the current behavior is wrong, I'm currently working to
>>> fix it asap. Our intention was to disable that only for non-committers.
>>>
>>> As a workaround, as a committer, you could manually add yourself (your
>>> GitHub username) to the whitelist of the SeedJob configuration:
>>> https://ci-beam.apache.org/job/beam_SeedJob/configure
>>> Then, your comment "Run Seed Job" will trigger the build. I've already
>>> manually triggered it for you that way.
>>>
>>> Of course, it will only work until the seed job gets executed - it will
>>> then override the whitelist with an empty one.
>>>
>>> [image: Selection_408.png]
>>>
>>> As a target solution, I'm planning to fetch the list of beam committers
>>> from LDAP and automatically add them to the whitelist above as a part of
>>> the seed job. I'll keep you updated about the progress.
>>>
>>> Regards,
>>> Damian
>>>
>>>
>>> On Wed, Jul 22, 2020 at 11:03 PM Ahmet Altay  wrote:
>>>
 +Damian Gadomski , it might be related to
 this change: https://github.com/apache/beam/pull/12319.

 /cc +Tyson Hamilton 

 On Wed, Jul 22, 2020 at 1:17 PM Udi Meiri  wrote:

> HI,
> I'm trying to test a groovy change but I can't seem to trigger the
> seed job. It worked yesterday so I'm not sure what changed.
>
> https://github.com/apache/beam/pull/12326
>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [VOTE] Make Apache Beam 2.24.0 the final release supporting Python 2.

[Chamikara Jayalath]

+1

On Thu, Jul 23, 2020 at 1:15 PM Brian Hulette  wrote:

> +1
>
> On Thu, Jul 23, 2020 at 1:05 PM Robert Bradshaw 
> wrote:
>
>> [X] +1: Remove Python 2 support in Apache Beam 2.25.0.
>>
>> According to our six-week release cadence, 2.24.0 (the last release to
>> support Python 2) will be cut mid-August, and the first release not
>> supporting Python 2 would be expected to land sometime in October. This
>> seems a reasonable timeline to me.
>>
>>
>> On Thu, Jul 23, 2020 at 12:53 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Please vote whether to make Apache Beam 2.24.0 the final release
>>> supporting Python 2 as follows.
>>>
>>> [ ] +1: Remove Python 2 support in Apache Beam 2.25.0.
>>> [ ] -1: Continue to support Python 2 in Apache Beam, and reconsider at a
>>> later date.
>>>
>>> The Beam community has pledged to sunset Python 2 support at some point
>>> in 2020[1,2]. A recent discussion[3] on dev@  proposes to outline a
>>> specific version after which Beam developers no longer have to maintain Py2
>>> support, which is a motivation for this vote.
>>>
>>> If this vote is approved we will announce Apache Beam 2.24.0 as our
>>> final release to support Python 2 and discontinue Python 2 support starting
>>> from 2.25.0 (inclusive).
>>>
>>> This is a procedural vote [4] that will follow the majority approval
>>> rules and will be open for at least 72 hours.
>>>
>>> Thanks,
>>> Valentyn
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/634f7346b607e779622d0437ed0eca783f474dea8976adf41556845b%40%3Cdev.beam.apache.org%3E
>>> [2] https://python3statement.org/
>>> [3]
>>> https://lists.apache.org/thread.html/r0d5c309a7e3107854f4892ccfeb1a17c0cec25dfce188678ab8df072%40%3Cdev.beam.apache.org%3E
>>> [4] https://www.apache.org/foundation/voting.html
>>>
>>>

Re: [VOTE] Make Apache Beam 2.24.0 the final release supporting Python 2.

[Ahmet Altay]

[X] +1: Remove Python 2 support in Apache Beam 2.25.0.

On Thu, Jul 23, 2020 at 1:05 PM Robert Bradshaw  wrote:

> [X] +1: Remove Python 2 support in Apache Beam 2.25.0.
>
> According to our six-week release cadence, 2.24.0 (the last release to
> support Python 2) will be cut mid-August, and the first release not
> supporting Python 2 would be expected to land sometime in October. This
> seems a reasonable timeline to me.
>
>
> On Thu, Jul 23, 2020 at 12:53 PM Valentyn Tymofieiev 
> wrote:
>
>> Hi everyone,
>>
>> Please vote whether to make Apache Beam 2.24.0 the final release
>> supporting Python 2 as follows.
>>
>> [ ] +1: Remove Python 2 support in Apache Beam 2.25.0.
>> [ ] -1: Continue to support Python 2 in Apache Beam, and reconsider at a
>> later date.
>>
>> The Beam community has pledged to sunset Python 2 support at some point
>> in 2020[1,2]. A recent discussion[3] on dev@  proposes to outline a
>> specific version after which Beam developers no longer have to maintain Py2
>> support, which is a motivation for this vote.
>>
>> If this vote is approved we will announce Apache Beam 2.24.0 as our final
>> release to support Python 2 and discontinue Python 2 support starting from
>> 2.25.0 (inclusive).
>>
>> This is a procedural vote [4] that will follow the majority approval
>> rules and will be open for at least 72 hours.
>>
>> Thanks,
>> Valentyn
>>
>> [1]
>> https://lists.apache.org/thread.html/634f7346b607e779622d0437ed0eca783f474dea8976adf41556845b%40%3Cdev.beam.apache.org%3E
>> [2] https://python3statement.org/
>> [3]
>> https://lists.apache.org/thread.html/r0d5c309a7e3107854f4892ccfeb1a17c0cec25dfce188678ab8df072%40%3Cdev.beam.apache.org%3E
>> [4] https://www.apache.org/foundation/voting.html
>>
>>

Re: [VOTE] Make Apache Beam 2.24.0 the final release supporting Python 2.

[Brian Hulette]

+1

On Thu, Jul 23, 2020 at 1:05 PM Robert Bradshaw  wrote:

> [X] +1: Remove Python 2 support in Apache Beam 2.25.0.
>
> According to our six-week release cadence, 2.24.0 (the last release to
> support Python 2) will be cut mid-August, and the first release not
> supporting Python 2 would be expected to land sometime in October. This
> seems a reasonable timeline to me.
>
>
> On Thu, Jul 23, 2020 at 12:53 PM Valentyn Tymofieiev 
> wrote:
>
>> Hi everyone,
>>
>> Please vote whether to make Apache Beam 2.24.0 the final release
>> supporting Python 2 as follows.
>>
>> [ ] +1: Remove Python 2 support in Apache Beam 2.25.0.
>> [ ] -1: Continue to support Python 2 in Apache Beam, and reconsider at a
>> later date.
>>
>> The Beam community has pledged to sunset Python 2 support at some point
>> in 2020[1,2]. A recent discussion[3] on dev@  proposes to outline a
>> specific version after which Beam developers no longer have to maintain Py2
>> support, which is a motivation for this vote.
>>
>> If this vote is approved we will announce Apache Beam 2.24.0 as our final
>> release to support Python 2 and discontinue Python 2 support starting from
>> 2.25.0 (inclusive).
>>
>> This is a procedural vote [4] that will follow the majority approval
>> rules and will be open for at least 72 hours.
>>
>> Thanks,
>> Valentyn
>>
>> [1]
>> https://lists.apache.org/thread.html/634f7346b607e779622d0437ed0eca783f474dea8976adf41556845b%40%3Cdev.beam.apache.org%3E
>> [2] https://python3statement.org/
>> [3]
>> https://lists.apache.org/thread.html/r0d5c309a7e3107854f4892ccfeb1a17c0cec25dfce188678ab8df072%40%3Cdev.beam.apache.org%3E
>> [4] https://www.apache.org/foundation/voting.html
>>
>>

Re: Jenkins trigger phrase "run seed job" not working?

[Damian Gadomski]

Yes, I thought that whitelisting apache organization will do the trick, but
apparently, it doesn't. Actually, it makes sense as we want to allow only
beam committers and not all apache committers. I don't know the
implications of membership in the apache github organization, but you for
instance are not there :) Neither is Ahmet.


Therefore there's nothing wrong with the Ghprb plugin, it correctly forbade
triggering. From my investigation, the "beam-committers" GitHub team (which
is under the apache org) is the list of people that should be allowed. But
firstly, you cant whitelist a team with Ghprb. There's a ticket for that,
open for 5 years . I
could implement that but, secondly, the team is secret. I can't even see
it. Even asfbot doesn't have permission to see it.

You may ask, how it worked before, because on the builds.apache.org somehow
only committers were allowed to trigger PR builds. It appeared that Infra
created a webhook relay. It's configured here

and
it filters out all the non-committers events. I wish I had known that
before as it was also the reason for different issues during the migration.
Anyway, it would be hard to use that mechanism in our case as we want to
configure it depending on the job.


There's a publicly available source of committers list - it's LDAP. I've
tested it and it allows anonymous connection and provides the list of the
committers as well as the github usernames. My current idea is to read this
from LDAP as a part of the seed job and configure the jobs with the apache
committers present on the ghprb whitelist.


Hope that I didn't miss anything ;) It isn't that easy to investigate that
kind of issues with my poor privileges ;)


Regards,

Damian


On Thu, Jul 23, 2020 at 6:52 PM Udi Meiri  wrote:

> Thanks Damian! I saw that the config also has this:
>   orgWhitelist(['apache'])
> Shouldn't that be enough to allow all Apache committers?
>
> I traced the code for the membership check here:
>
> https://github.com/jenkinsci/ghprb-plugin/blob/4e86ed47a96a01eeaa51a479ff604252109635f6/src/main/java/org/jenkinsci/plugins/ghprb/GhprbGitHub.java#L27
> Is there a way to see these logs?
>
>
> On Thu, Jul 23, 2020 at 7:08 AM Damian Gadomski <
> damian.gadom...@polidea.com> wrote:
>
>> Hi,
>>
>> You are right, the current behavior is wrong, I'm currently working to
>> fix it asap. Our intention was to disable that only for non-committers.
>>
>> As a workaround, as a committer, you could manually add yourself (your
>> GitHub username) to the whitelist of the SeedJob configuration:
>> https://ci-beam.apache.org/job/beam_SeedJob/configure
>> Then, your comment "Run Seed Job" will trigger the build. I've already
>> manually triggered it for you that way.
>>
>> Of course, it will only work until the seed job gets executed - it will
>> then override the whitelist with an empty one.
>>
>> [image: Selection_408.png]
>>
>> As a target solution, I'm planning to fetch the list of beam committers
>> from LDAP and automatically add them to the whitelist above as a part of
>> the seed job. I'll keep you updated about the progress.
>>
>> Regards,
>> Damian
>>
>>
>> On Wed, Jul 22, 2020 at 11:03 PM Ahmet Altay  wrote:
>>
>>> +Damian Gadomski , it might be related to
>>> this change: https://github.com/apache/beam/pull/12319.
>>>
>>> /cc +Tyson Hamilton 
>>>
>>> On Wed, Jul 22, 2020 at 1:17 PM Udi Meiri  wrote:
>>>
 HI,
 I'm trying to test a groovy change but I can't seem to trigger the seed
 job. It worked yesterday so I'm not sure what changed.

 https://github.com/apache/beam/pull/12326

Re: [VOTE] Make Apache Beam 2.24.0 the final release supporting Python 2.

[Robert Bradshaw]

[X] +1: Remove Python 2 support in Apache Beam 2.25.0.

According to our six-week release cadence, 2.24.0 (the last release to
support Python 2) will be cut mid-August, and the first release not
supporting Python 2 would be expected to land sometime in October. This
seems a reasonable timeline to me.


On Thu, Jul 23, 2020 at 12:53 PM Valentyn Tymofieiev 
wrote:

> Hi everyone,
>
> Please vote whether to make Apache Beam 2.24.0 the final release
> supporting Python 2 as follows.
>
> [ ] +1: Remove Python 2 support in Apache Beam 2.25.0.
> [ ] -1: Continue to support Python 2 in Apache Beam, and reconsider at a
> later date.
>
> The Beam community has pledged to sunset Python 2 support at some point in
> 2020[1,2]. A recent discussion[3] on dev@  proposes to outline a specific
> version after which Beam developers no longer have to maintain Py2 support,
> which is a motivation for this vote.
>
> If this vote is approved we will announce Apache Beam 2.24.0 as our final
> release to support Python 2 and discontinue Python 2 support starting from
> 2.25.0 (inclusive).
>
> This is a procedural vote [4] that will follow the majority approval
> rules and will be open for at least 72 hours.
>
> Thanks,
> Valentyn
>
> [1]
> https://lists.apache.org/thread.html/634f7346b607e779622d0437ed0eca783f474dea8976adf41556845b%40%3Cdev.beam.apache.org%3E
> [2] https://python3statement.org/
> [3]
> https://lists.apache.org/thread.html/r0d5c309a7e3107854f4892ccfeb1a17c0cec25dfce188678ab8df072%40%3Cdev.beam.apache.org%3E
> [4] https://www.apache.org/foundation/voting.html
>
>

[VOTE] Make Apache Beam 2.24.0 the final release supporting Python 2.

[Valentyn Tymofieiev]

Hi everyone,

Please vote whether to make Apache Beam 2.24.0 the final release supporting
Python 2 as follows.

[ ] +1: Remove Python 2 support in Apache Beam 2.25.0.
[ ] -1: Continue to support Python 2 in Apache Beam, and reconsider at a
later date.

The Beam community has pledged to sunset Python 2 support at some point in
2020[1,2]. A recent discussion[3] on dev@  proposes to outline a specific
version after which Beam developers no longer have to maintain Py2 support,
which is a motivation for this vote.

If this vote is approved we will announce Apache Beam 2.24.0 as our final
release to support Python 2 and discontinue Python 2 support starting from
2.25.0 (inclusive).

This is a procedural vote [4] that will follow the majority approval rules
and will be open for at least 72 hours.

Thanks,
Valentyn

[1]
https://lists.apache.org/thread.html/634f7346b607e779622d0437ed0eca783f474dea8976adf41556845b%40%3Cdev.beam.apache.org%3E
[2] https://python3statement.org/
[3]
https://lists.apache.org/thread.html/r0d5c309a7e3107854f4892ccfeb1a17c0cec25dfce188678ab8df072%40%3Cdev.beam.apache.org%3E
[4] https://www.apache.org/foundation/voting.html

Re: Python2.7 Beam End-of-Life Date

[Valentyn Tymofieiev]

I have reviewed Slack, Twitter and User@ channels [1-3] we used to
communicate our consideration to make 2.23.0 a final release supporting
Py2.

I did not see any objections or negative feedback there.

I suggest we hold a VOTE to decide whether 2.24.0 should be the final
release supporting Py2. I can send the voting email.

[1]
https://app.slack.com/client/T4S1WH2J3/CBDNLQZM1/thread/C9H0YNP3P-1592524218.050700
[2] https://twitter.com/ApacheBeam/status/1273760375570194432
[3]
https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E

On Thu, Jul 23, 2020 at 10:57 AM Robert Bradshaw 
wrote:

> The release cut date for 2.24 is in a couple of weeks; if this is the last
> release supporting 2.7 we should make the call and announce it soon.
>
> On Mon, Jul 6, 2020 at 2:26 PM Robert Bradshaw 
> wrote:
>
>> Note that just because Beam drops 2.x support in new releases doesn't
>> mean that the old releases won't continue to work. One can even use an
>> expansion service to run 2.x transforms (on an older version of Beam)
>> within a Python 3 pipeline running the newest version of Beam until
>> such a time that (possibly incrementally) the dependent libraries
>> catch up, if it comes to that. Not that this will be ideal.
>>
>> Now that the 2.23 release branch has been cut, we *could* remove 2.7
>> support if we will actually plan to remove it in 2.24. (One concern,
>> however, is how tied the release testing infrastructure is to
>> mainline...) However, other than Chad's response (the VFX / Animation
>> isn't quite there yet) it seems we still don't have a very strong
>> signal either direction. Pypi downloads are still hovering around 50%.
>> Did we hear anything back from twitter/slack?
>>
>> My inclination would be to publish loudly that 2.24 would be the last
>> Beam to support Python 2.7 (this is pushing it back one release), so
>> if you're still stuck on it make sure it has everything you need by
>> the release cut date (which is still in the future).
>>
>>
>> On Mon, Jul 6, 2020 at 9:52 AM David Yan  wrote:
>> >
>> > +1 for removing Python 2.7 support sooner than later.
>> >
>> > I recently added a small feature in Beam Python and I found that having
>> to write code that worked with Python2 was quite awkward and time consuming
>> (needing to make sure code works for both 2 and 3 and doubling the Jenkins
>> running time), and I can imagine that may hinder or even discourage new
>> contributions.
>> >
>> > On Thu, Jun 18, 2020 at 5:12 PM Ahmet Altay  wrote:
>> >>
>> >> Good to hear from you and good to hear the news. Release branch cut
>> date for 2.24 is 8/12.
>> >>
>> >> On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova 
>> wrote:
>> >>>
>> >>> Hi all,
>> >>> Sorry I've been AWOL.  I've been pulled in a number of different
>> directions.
>> >>>
>> >>> We are still increasing our use of Beam, and I have it on my todo
>> list to reach out to Kenneth about how Beam could be expanded into the VFX
>> / Animation industries.
>> >>>
>> >>> Our industry uses a number of specialized applications with embedded
>> python interpreters.  We run Beam inside these interpreters, so we're
>> waiting for them to switch to python3.
>> >>>
>> >>> Here's the status report for python3 adoption in our key applications:
>> >>>
>> >>> Maya:  In Beta
>> >>> Houdini:  Released
>> >>> Nuke: In Beta
>> >>> Katana:  Not started (Alpha?)
>> >>>
>> >>> I hate to be the one holding the project back, and I understand if
>> you all ultimately decide it's untenable to wait any longer.  The good news
>> is 3 out of 4 applications should be ready in the next 2-3 months.  I can
>> do some investigation into what workarounds might look like for Katana, or
>> maybe we can use the Beta version once python3 support arrives, which would
>> move our schedule forward.
>> >>>
>> >>> When would 2.24 release?
>> >>>
>> >>> -chad
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay  wrote:
>> 
>>  OK, tweeted the message. Could you share on Slack?
>> 
>>  On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>> >
>> > Alright, let's publish the message on Twitter and echo on Slack
>> once that's done.
>> > Thank you!
>> >
>> > On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay 
>> wrote:
>> >>
>> >> That sounds reasonable to me. I agree, it is better to get
>> specific reasons rather than a yes/no answer.
>> >>
>> >> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>> >>>
>> >>> After thinking about it for a bit, I am not sure whether a poll
>> would be actionable. For example, if 1000 users provide a positive response
>> and 5 users provide a negative response, how do we interpret that and
>> where do we draw a line? How about instead we encourage users to reach out,
>> and tell what is not working for them? For exa

Contributor permission for Beam Jira tickets

[Varun Sharma]

Hi,

This is Varun from THG. I have created a couple of tickets related to
improvements in Beam Java SDK. Could someone add me as a contributor
for Beam's Jira issue tracker? I would like to assign tickets for my work.

My JIRA id- varun.sharma


Regards
Varun Sharma

Re: ReadFromKafka returns error - RuntimeError: cannot encode a null byte[]

[ayush sharma]

I tried enabling auto commit using flink and direct runners individually
using the code below.
Irrespective of environment-type ["DOCKER", "LOOPBACK"] for flink runner,
the pipeline crashes with ERROR: java.lang.UnsupportedOperationException:
The ActiveBundle does not have a registered bundle checkpoint handler.
And direct runner (no args in PipelineOptions()) does not respond to the
reception of the published kafka message.

# USAGE:
# 1. python sof_kafka_read_v2.py --runner flink
# 2. python sof_kafka_read_v2.py --runner flink --environment-type DOCKER
# 3. python sof_kafka_read_v2.py

import apache_beam as beam
from apache_beam.io.external.kafka import ReadFromKafka, WriteToKafka
from apache_beam.options.pipeline_options import PipelineOptions,
StandardOptions, SetupOptions
import argparse

if __name__ == '__main__':

parser = argparse.ArgumentParser()

parser.add_argument(
'--runner',
type=str,
default="direct",
)
parser.add_argument(
'--environment-type',
type=str,
default="LOOPBACK",
)

args = parser.parse_args()

runner = args.runner
environment_type = args.environment_type

if runner == "direct":
options = PipelineOptions()
elif runner == "flink":
options = PipelineOptions([
"--runner=FlinkRunner",
"--flink_version=1.9",
"--flink_master=localhost:8081",
"--environment_type=" + environment_type,
"--job_name=try_kafka",
"--streaming",
])

options.view_as(SetupOptions).save_main_session = True
options.view_as(StandardOptions).streaming = True

pipeline = beam.Pipeline(options=options)

result = (
pipeline

| "Read from kafka" >> ReadFromKafka(
consumer_config={
"bootstrap.servers": 'localhost:9092',
"enable.auto.commit": 'true',
},
topics=['mytopic'],
expansion_service='localhost:8097',
)

| "print" >> beam.Map(print)
)

res = pipeline.run()
res.wait_until_finish()



On Wed, Jul 22, 2020 at 7:39 PM Robert Bradshaw  wrote:

> On Sat, Jul 18, 2020 at 12:08 PM Chamikara Jayalath 
> wrote:
>
>>
>>
>> On Fri, Jul 17, 2020 at 10:04 PM ayush sharma <1705ay...@gmail.com>
>> wrote:
>>
>>> Thank you guys for the reply. I am really stuck and could not proceed
>>> further.
>>> Yes, the previous trial published message had null key.
>>> But when I send key:value pair through producer using
>>>
>>> ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
>>> mytopic --property *"parse.key=true" --property "key.separator=:"*
>>> > tryKey:tryValue
>>>
>>> I do not get any error but beam does not print the received message.
>>> Here is how my pipeline looks like,
>>> result = (
>>> pipeline
>>>
>>> | "Read from kafka" >> ReadFromKafka(
>>> consumer_config={
>>> "bootstrap.servers": 'localhost:9092',
>>> },
>>> topics=['mytopic'],
>>> expansion_service='localhost:8097',
>>>
>>> | "print" >> beam.Map(print)
>>> )
>>>
>>>
>> I suspect DirectRunner in LOOPBACK mode might not be working for
>> cross-language transforms today.
>>
>
> When running a Streaming pipeline, the DirectRuner falls back to the old
> runner that does not support cross-language.
> https://issues.apache.org/jira/browse/BEAM-7514
>
> Please note that cross-language transforms framework is fairly new [1] and
>> we are adding support for various runners and environment configurations.
>> Can you try with Flink in DOCKER mode ?
>>
>>
>>> If this is not the way we make beam and kafka communicate then please
>>> share a working example which showcases how a message published in kafka
>>> gets received by beam while streaming.
>>>
>>
>> I'm adding an example but I've only tested this with Dataflow yet. I hope
>> to test that example for more runners and add additional instructions
>> there.
>> https://github.com/apache/beam/pull/12188
>>
>> Thanks,
>> Cham
>>
>> [1] https://beam.apache.org/roadmap/connectors-multi-sdk/
>>
>>>
>>> Regards,
>>> Ayush Sharma
>>>
>>> On Fri, Jul 17, 2020 at 11:39 PM Chamikara Jayalath <
>>> chamik...@google.com> wrote:
>>>
 Yes, seems like this is due to the key being null. XLang KafkaIO has to
 be updated to support this. You should not run into this error if you
 publish keys and values that are not null.




 On Fri, Jul 17, 2020 at 8:04 PM Luke Cwik  wrote:

> +dev 
>
> On Fri, Jul 17, 2020 at 8:03 PM Luke Cwik  wrote:
>
>> +Heejong Lee  +Chamikara Jayalath
>> 
>>
>> Do you know if your trial record has an empty key or value?
>> If so, then you hit a bug and it seems as though there was a miss
>> supporting this usecase.
>>
>> Heejong and Cham,
>> It looks like the Javadoc for ByteArrayDeserializer and other
>> Deserializers can return nu

Re: Python2.7 Beam End-of-Life Date

[Robert Bradshaw]

The release cut date for 2.24 is in a couple of weeks; if this is the last
release supporting 2.7 we should make the call and announce it soon.

On Mon, Jul 6, 2020 at 2:26 PM Robert Bradshaw  wrote:

> Note that just because Beam drops 2.x support in new releases doesn't
> mean that the old releases won't continue to work. One can even use an
> expansion service to run 2.x transforms (on an older version of Beam)
> within a Python 3 pipeline running the newest version of Beam until
> such a time that (possibly incrementally) the dependent libraries
> catch up, if it comes to that. Not that this will be ideal.
>
> Now that the 2.23 release branch has been cut, we *could* remove 2.7
> support if we will actually plan to remove it in 2.24. (One concern,
> however, is how tied the release testing infrastructure is to
> mainline...) However, other than Chad's response (the VFX / Animation
> isn't quite there yet) it seems we still don't have a very strong
> signal either direction. Pypi downloads are still hovering around 50%.
> Did we hear anything back from twitter/slack?
>
> My inclination would be to publish loudly that 2.24 would be the last
> Beam to support Python 2.7 (this is pushing it back one release), so
> if you're still stuck on it make sure it has everything you need by
> the release cut date (which is still in the future).
>
>
> On Mon, Jul 6, 2020 at 9:52 AM David Yan  wrote:
> >
> > +1 for removing Python 2.7 support sooner than later.
> >
> > I recently added a small feature in Beam Python and I found that having
> to write code that worked with Python2 was quite awkward and time consuming
> (needing to make sure code works for both 2 and 3 and doubling the Jenkins
> running time), and I can imagine that may hinder or even discourage new
> contributions.
> >
> > On Thu, Jun 18, 2020 at 5:12 PM Ahmet Altay  wrote:
> >>
> >> Good to hear from you and good to hear the news. Release branch cut
> date for 2.24 is 8/12.
> >>
> >> On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova 
> wrote:
> >>>
> >>> Hi all,
> >>> Sorry I've been AWOL.  I've been pulled in a number of different
> directions.
> >>>
> >>> We are still increasing our use of Beam, and I have it on my todo list
> to reach out to Kenneth about how Beam could be expanded into the VFX /
> Animation industries.
> >>>
> >>> Our industry uses a number of specialized applications with embedded
> python interpreters.  We run Beam inside these interpreters, so we're
> waiting for them to switch to python3.
> >>>
> >>> Here's the status report for python3 adoption in our key applications:
> >>>
> >>> Maya:  In Beta
> >>> Houdini:  Released
> >>> Nuke: In Beta
> >>> Katana:  Not started (Alpha?)
> >>>
> >>> I hate to be the one holding the project back, and I understand if you
> all ultimately decide it's untenable to wait any longer.  The good news is
> 3 out of 4 applications should be ready in the next 2-3 months.  I can do
> some investigation into what workarounds might look like for Katana, or
> maybe we can use the Beta version once python3 support arrives, which would
> move our schedule forward.
> >>>
> >>> When would 2.24 release?
> >>>
> >>> -chad
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay  wrote:
> 
>  OK, tweeted the message. Could you share on Slack?
> 
>  On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> >
> > Alright, let's publish the message on Twitter and echo on Slack once
> that's done.
> > Thank you!
> >
> > On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay 
> wrote:
> >>
> >> That sounds reasonable to me. I agree, it is better to get specific
> reasons rather than a yes/no answer.
> >>
> >> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> >>>
> >>> After thinking about it for a bit, I am not sure whether a poll
> would be actionable. For example, if 1000 users provide a positive response
> and 5 users provide a negative response, how do we interpret that and
> where do we draw a line? How about instead we encourage users to reach out,
> and tell what is not working for them? For example:
> >>>
> >>> Beam is considering making 2.23.0 a final release supporting Py2.
> If you are not able to switch to Python 3, please let us know why: [short
> link to user@ thread] [1].
> >>>
> >>> Thoughts?
> >>>
> >>> [1]
> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
> >>>
> >>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay 
> wrote:
> 
> 
> 
>  On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> >
> > I have reached out to user@ for feedback on Python 3
> migration[1].
> 
> 
>  Maybe also ask on slack? There are quite a bit of users there as
> we

Re: [VOTE] Release 2.23.0, release candidate #2

[Robert Bradshaw]

+1 (binding)

I validated the hashes and signatures of all the release artifacts, and
that the source tarball matches github
at 5df6e7949629799c46d227171281364144149f5d. I also verified that the diff
from the last RC contains what is expected and ran some basic pipelines
against a Python 3 wheel in a fresh virtual environment. All looks good.

On Thu, Jul 23, 2020 at 9:47 AM Chamikara Jayalath 
wrote:

> +1 (non-binding)
> Tested Java quickstart and an x-lang pipeline.
>
> Thanks,
> Cham
>
> On Wed, Jul 22, 2020 at 7:34 PM Ahmet Altay  wrote:
>
>> +1 - I validated py3 quickstarts.
>>
>> On Wed, Jul 22, 2020 at 6:21 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Please review and vote on the release candidate #2 for the version
>>> 2.23.0, as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> The complete staging area is available for your review, which includes:
>>> * JIRA release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org [2],
>>> which is signed with the key with fingerprint 1DF50603225D29A4 [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.23.0-RС2" [5],
>>> * website pull request listing the release [6], publishing the API
>>> reference manual [7], and the blog post [8].
>>> * Java artifacts were built with Maven 3.6.0 and Oracle JDK
>>> 1.8.0_201-b09 .
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2].
>>> * Validation sheet with a tab for 2.23.0 release to help with validation
>>> [9].
>>> * Docker images published to Docker Hub [10].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>> Release manager.
>>>
>>> [1]
>>> https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12347145
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.23.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1106/
>>> [5] https://github.com/apache/beam/tree/v2.23.0-RC2
>>> [6] https://github.com/apache/beam/pull/12212
>>> [7] https://github.com/apache/beam-site/pull/605
>>> [8] https://github.com/apache/beam/pull/12213
>>> [9]
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=596347973
>>> [10] https://hub.docker.com/search?q=apache%2Fbeam&type=image
>>>
>>

Re: Jenkins trigger phrase "run seed job" not working?

[Udi Meiri]

Thanks Damian! I saw that the config also has this:
  orgWhitelist(['apache'])
Shouldn't that be enough to allow all Apache committers?

I traced the code for the membership check here:
https://github.com/jenkinsci/ghprb-plugin/blob/4e86ed47a96a01eeaa51a479ff604252109635f6/src/main/java/org/jenkinsci/plugins/ghprb/GhprbGitHub.java#L27
Is there a way to see these logs?


On Thu, Jul 23, 2020 at 7:08 AM Damian Gadomski 
wrote:

> Hi,
>
> You are right, the current behavior is wrong, I'm currently working to fix
> it asap. Our intention was to disable that only for non-committers.
>
> As a workaround, as a committer, you could manually add yourself (your
> GitHub username) to the whitelist of the SeedJob configuration:
> https://ci-beam.apache.org/job/beam_SeedJob/configure
> Then, your comment "Run Seed Job" will trigger the build. I've already
> manually triggered it for you that way.
>
> Of course, it will only work until the seed job gets executed - it will
> then override the whitelist with an empty one.
>
> [image: Selection_408.png]
>
> As a target solution, I'm planning to fetch the list of beam committers
> from LDAP and automatically add them to the whitelist above as a part of
> the seed job. I'll keep you updated about the progress.
>
> Regards,
> Damian
>
>
> On Wed, Jul 22, 2020 at 11:03 PM Ahmet Altay  wrote:
>
>> +Damian Gadomski , it might be related to
>> this change: https://github.com/apache/beam/pull/12319.
>>
>> /cc +Tyson Hamilton 
>>
>> On Wed, Jul 22, 2020 at 1:17 PM Udi Meiri  wrote:
>>
>>> HI,
>>> I'm trying to test a groovy change but I can't seem to trigger the seed
>>> job. It worked yesterday so I'm not sure what changed.
>>>
>>> https://github.com/apache/beam/pull/12326
>>>
>>>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Contributor permission for Beam Jira tickets

[Ahmet Altay]

Done, added rworley as a contributor. Welcome.

On Thu, Jul 23, 2020 at 9:29 AM Worley, Ryan 
wrote:

> Hi,
>
> I have created BEAM-10564 and I would like to assign it to myself.  Can
> someone please add me as a contributor for Beam's Jira issue tracker?  This
> is the first of two pull requests that I plan to submit with minor
> improvements around working with Avro and BigQuery.
>
> Thanks,
> Ryan
>
> NOTICE:
>
> This message, and any attachments, contain(s) information that may be
> confidential or protected by privilege from disclosure and is intended only
> for the individual or entity named above. No one else may disclose, copy,
> distribute or use the contents of this message for any purpose. Its
> unauthorized use, dissemination or duplication is strictly prohibited and
> may be unlawful. If you receive this message in error or you otherwise are
> not an authorized recipient, please immediately delete the message and any
> attachments and notify the sender.
>

Re: [BROKEN] Please add "Fix Version" when resolving or closing Jiras

[Brian Hulette]

Yeah same for me. Not a huge deal.. I just noticed that the jira UI isn't
rendering references to my closed tickets with a strikethrough, and I'd
like to distinguish others as "won't fix" or "obsolete". It should be easy
enough to backfill these once this is sorted out though.

Brian

On Thu, Jul 23, 2020 at 2:01 AM Maximilian Michels  wrote:

> I can close/resolve but it will show "Resolution: Unresolved":
>
> On 23.07.20 02:30, Brian Hulette wrote:
>
> Is setting the Resolution broken as well? I realized I've been closing
> jiras with Resolution "Unresolved" and I can't actually change it to
> "Fixed".
>
> On Tue, Jul 21, 2020 at 7:19 AM Maximilian Michels  wrote:
>
>> Also, a friendly reminder to always close the JIRA issue after merging a
>> fix. It's easy to forget.
>>
>> On 20.07.20 21:04, Kenneth Knowles wrote:
>> > Hi all,
>> >
>> > In working on our Jira automation, I've messed up our Jira workflow. It
>> > will no longer prompt you to fill in "Fix Version" when you resolve or
>> > close an issue. I will be working with infra to restore this. In the
>> > meantime, please try to remember to add a Fix Version to each issue
>> that
>> > you close, so that we get automated detailed release notes.
>> >
>> > Kenn
>>
>

Re: [VOTE] Release 2.23.0, release candidate #2

[Chamikara Jayalath]

+1 (non-binding)
Tested Java quickstart and an x-lang pipeline.

Thanks,
Cham

On Wed, Jul 22, 2020 at 7:34 PM Ahmet Altay  wrote:

> +1 - I validated py3 quickstarts.
>
> On Wed, Jul 22, 2020 at 6:21 PM Valentyn Tymofieiev 
> wrote:
>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #2 for the version
>> 2.23.0, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org [2],
>> which is signed with the key with fingerprint 1DF50603225D29A4 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.23.0-RС2" [5],
>> * website pull request listing the release [6], publishing the API
>> reference manual [7], and the blog post [8].
>> * Java artifacts were built with Maven 3.6.0 and Oracle JDK 1.8.0_201-b09
>> .
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2].
>> * Validation sheet with a tab for 2.23.0 release to help with validation
>> [9].
>> * Docker images published to Docker Hub [10].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> Release manager.
>>
>> [1]
>> https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12347145
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.23.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1106/
>> [5] https://github.com/apache/beam/tree/v2.23.0-RC2
>> [6] https://github.com/apache/beam/pull/12212
>> [7] https://github.com/apache/beam-site/pull/605
>> [8] https://github.com/apache/beam/pull/12213
>> [9]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=596347973
>> [10] https://hub.docker.com/search?q=apache%2Fbeam&type=image
>>
>

Contributor permission for Beam Jira tickets

[Worley, Ryan]

Hi,

I have created BEAM-10564 and I would like to assign it to myself.  Can someone 
please add me as a contributor for Beam's Jira issue tracker?  This is the 
first of two pull requests that I plan to submit with minor improvements around 
working with Avro and BigQuery.

Thanks,
Ryan

NOTICE:

This message, and any attachments, contain(s) information that may be 
confidential or protected by privilege from disclosure and is intended only for 
the individual or entity named above. No one else may disclose, copy, 
distribute or use the contents of this message for any purpose. Its 
unauthorized use, dissemination or duplication is strictly prohibited and may 
be unlawful. If you receive this message in error or you otherwise are not an 
authorized recipient, please immediately delete the message and any attachments 
and notify the sender.

Re: Jenkins trigger phrase "run seed job" not working?

[Damian Gadomski]

Hi,

You are right, the current behavior is wrong, I'm currently working to fix
it asap. Our intention was to disable that only for non-committers.

As a workaround, as a committer, you could manually add yourself (your
GitHub username) to the whitelist of the SeedJob configuration:
https://ci-beam.apache.org/job/beam_SeedJob/configure
Then, your comment "Run Seed Job" will trigger the build. I've already
manually triggered it for you that way.

Of course, it will only work until the seed job gets executed - it will
then override the whitelist with an empty one.

[image: Selection_408.png]

As a target solution, I'm planning to fetch the list of beam committers
from LDAP and automatically add them to the whitelist above as a part of
the seed job. I'll keep you updated about the progress.

Regards,
Damian


On Wed, Jul 22, 2020 at 11:03 PM Ahmet Altay  wrote:

> +Damian Gadomski , it might be related to
> this change: https://github.com/apache/beam/pull/12319.
>
> /cc +Tyson Hamilton 
>
> On Wed, Jul 22, 2020 at 1:17 PM Udi Meiri  wrote:
>
>> HI,
>> I'm trying to test a groovy change but I can't seem to trigger the seed
>> job. It worked yesterday so I'm not sure what changed.
>>
>> https://github.com/apache/beam/pull/12326
>>
>>

Re: [BROKEN] Please add "Fix Version" when resolving or closing Jiras

[Maximilian Michels]

I can close/resolve but it will show "Resolution: Unresolved":

On 23.07.20 02:30, Brian Hulette wrote:
Is setting the Resolution broken as well? I realized I've been closing 
jiras with Resolution "Unresolved" and I can't actually change it to 
"Fixed".


On Tue, Jul 21, 2020 at 7:19 AM Maximilian Michels > wrote:


Also, a friendly reminder to always close the JIRA issue after
merging a
fix. It's easy to forget.

On 20.07.20 21:04, Kenneth Knowles wrote:
> Hi all,
>
> In working on our Jira automation, I've messed up our Jira
workflow. It
> will no longer prompt you to fill in "Fix Version" when you
resolve or
> close an issue. I will be working with infra to restore this. In
the
> meantime, please try to remember to add a Fix Version to each
issue that
> you close, so that we get automated detailed release notes.
>
> Kenn