Re: [DISCUSS] Portability representation of schemas

[Brian Hulette]

If we want to have a Pipeline level registry, we could add it to Components
[1].

message Components {
  ...
  map logical_types;
}

And in FieldType reference the logical types by id:
oneof field_type {
  AtomicType atomic_type;
  ArrayType array_type;
  ...
  string logical_type_id;// was LogicalType logical_type;
}

I'm not sure I like this idea though. The reason we started discussing a
"registry" was just to separate the SDK-specific bits from the
representation type, and this doesn't accomplish that, it just de-dupes
logical types used
across the pipeline.

I think instead I'd rather just come back to the message we have now in the
doc, used directly in FieldType's oneof:

message LogicalType {
  FieldType representation = 1;
  string logical_urn = 2;
  bytes logical_payload = 3;
}

We can have a URN for SDK-specific types (user type aliases), like
"beam:logical:javasdk", and the logical_payload could itself be a protobuf
with attributes of 1) a serialized class and 2/3) to/from functions. For
truly portable types it would instead have a well-known URN and optionally
a logical_payload with some agreed-upon representation of parameters.

It seems like maybe SdkFunctionSpec/Environment should be used for this
somehow, but I can't find a good example of this in the Runner API to use
as a model. For example, what we're trying to accomplish is basically the
same as Java custom coders vs. standard coders. But that is accomplished
with a magic "javasdk" URN, as I suggested here, not with Environment
[2,3]. There is a "TODO: standardize such things" where that URN is
defined, is it possible that Environment is that standard and just hasn't
been utilized for custom coders yet?

Brian

[1]
https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L54
[2]
https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L542
[3]
https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java#L121

On Tue, Jun 4, 2019 at 2:24 PM Brian Hulette  wrote:

> Yeah that's what I meant. It does seem logical reasonable to scope any
> registry by pipeline and not by PCollection. Then it seems we would want
> the entire LogicalType (including the `FieldType representation` field) as
> the value type, and not just LogicalTypeConversion. Otherwise we're
> separating the representations from the conversions, and duplicating the
> representations. You did say a "registry of logical types", so maybe that
> is what you meant.
>
> Brian
>
> On Tue, Jun 4, 2019 at 1:21 PM Reuven Lax  wrote:
>
>>
>>
>> On Tue, Jun 4, 2019 at 9:20 AM Brian Hulette  wrote:
>>
>>>
>>>
>>> On Mon, Jun 3, 2019 at 10:04 PM Reuven Lax  wrote:
>>>


 On Mon, Jun 3, 2019 at 12:27 PM Brian Hulette 
 wrote:

> > It has to go into the proto somewhere (since that's the only way
> the SDK can get it), but I'm not sure they should be considered integral
> parts of the type.
> Are you just advocating for an approach where any SDK-specific
> information is stored outside of the Schema message itself so that Schema
> really does just represent the type? That seems reasonable to me, and
> alleviates my concerns about how this applies to columnar encodings a bit
> as well.
>

 Yes, that's exactly what I'm advocating.


>
> We could lift all of the LogicalTypeConversion messages out of the
> Schema and the LogicalType like this:
>
> message SchemaCoder {
>   Schema schema = 1;
>   LogicalTypeConversion root_conversion = 2;
>   map attribute_conversions = 3; //
> only necessary for user type aliases, portable logical types by definition
> have nothing SDK-specific
> }
>

 I'm not sure what the map is for? I think we have status quo wihtout it.

>>>
>>> My intention was that the SDK-specific information (to/from functions)
>>> for any nested fields that are themselves user type aliases would be stored
>>> in this map. That was the motivation for my next question, if we don't
>>> allow user types to be nested within other user types we may not need it.
>>>
>>
>> Oh, is this meant to contain the ids of all the logical types in this
>> schema? If so I don't think SchemaCoder is the right place for this. Any
>> "registry" of logical types should be global to the pipeline, not scoped to
>> a single PCollection IMO.
>>
>>
>>> I may be missing your meaning - but I think we currently only have
>>> status quo without this map in the Java SDK because Schema.LogicalType is
>>> just an interface that must be implemented. It's appropriate for just
>>> portable logical types, not user-type aliases. Note I've adopted Kenn's
>>> terminology where portable logical type is a type that can be identified by
>>> just a URN and maybe some parameters, while a user type alias needs some
>>> SDK specific 

Re: [VOTE] Release 2.13.0, release candidate #2

[Ankur Goenka]

Thanks Kenn, for identifying the issue.
If no artifacts affected artifacts are published then we should be good.
Let me know if we need to make any changes in 2.13.0

On Wed, Jun 5, 2019 at 10:53 AM Kenneth Knowles  wrote:

> Just discovered a potentially serious issue that was present during this
> RC: https://issues.apache.org/jira/browse/BEAM-7493. So far I have not
> discovered a truly user-facing impact, and example validation succeeded,
> but I wanted to alert the list.
>
> Summary: When rendering a published pom.xml the dependencies are always
> the path concatenated with dashes even when that is not the
> correct artifactId. For example sdks/java/testing/test-utils are resolved
> in the pom to beam-sdks-java-testing-test-utils. This does not exist; it
> manually sets the name to beam-sdks-java-test-utils, omitting the
> extraneous `testing` directory that exists only for taxonomy.
>
> There are a few other modules that manually set their archive name. From
> what I can tell, each of these is either (a) not published or (b) not
> depended upon. I am still checking.
>
> Kenn
>
>
> On Wed, Jun 5, 2019 at 9:22 AM Thomas Weise  wrote:
>
>> +1 and I think all of that can be covered with JIRA.
>>
>> Irrespective the release manager still needs to pay attention to the
>> communication on the VOTE thread.
>>
>> On Wed, Jun 5, 2019 at 9:19 AM Ahmet Altay  wrote:
>>
>>> Checking that JIRA link sounds reasonable as long as we can agree that
>>> it is single source of truth for cherry pick requests. I also agree with
>>> Cham, requests need to come with a reason.
>>>
>>> On Wed, Jun 5, 2019 at 7:38 AM Ismaël Mejía  wrote:
>>>
 I don't think we need anything fancier or marking even as Blocker some
 of this stuff, would not be enough just to monitor that [1] has no
 issues? (of course if the interested party has not put the fix version
 to the current ongoing vote one this is a mistake).

 [1]
 https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.13.0

 On Wed, Jun 5, 2019 at 4:23 PM Chamikara Jayalath 
 wrote:
 >
 >
 >
 > On Tue, Jun 4, 2019 at 5:02 PM Ahmet Altay  wrote:
 >>
 >> I would suggest have a single way of tracking cherry pick request to
 an RC. Currently we use emails on the RC thread, open PRs, and Jiras tagged
 for the release. This is confusing the person doing the release while they
 are juggling multiple things. How about we ask all cherry pick requests to
 have a JIRA filed against that release and marked as blockers?
 >
 >
 > I agree with this and with what Ankur said. Release blockers should
 be more explicit and should come with a reason. After voting thread start,
 I would say this should include a mail to the voting thread as well as a
 blocking JIRA. Other PRs opened against the branch may or may not get
 merged at the discretion of the release manager.
 >
 > Thanks,
 > Cham
 >
 >>
 >>
 >> On Tue, Jun 4, 2019 at 1:05 PM Ankur Goenka 
 wrote:
 >>>
 >>> That makes sense.
 >>> I would also like to add that the corresponding PR should be added
 to an open blocking Jira for the release to keep a single source to check.
 >>>
 >>> On Tue, Jun 4, 2019 at 12:15 PM Kenneth Knowles 
 wrote:
 
  I would actually suggest that the following search needs to be
 triaged to zero before cutting an RC:
 https://github.com/apache/beam/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+base%3Arelease-2.13.0
 .
 
  On Tue, Jun 4, 2019 at 11:17 AM Ankur Goenka 
 wrote:
 >
 > Sorry, I missed the comment for not including weekend's to 72
 hours voting period.
 >
 > I meant to update the blog post
 https://github.com/apache/beam/pull/8667/files once we have finalized
 the RC so that it can be consistent. Please add any comments to PR and I
 can incorporate them.
 >
 > As we did not go for 3rd RC and
 https://github.com/apache/beam/pull/8714 was not blocking the 2.13
 release, I went with the release.
 >
 > I have released the maven artifacts for beam. So I suppose, we
 can not do another RC for 2.13.0.
 > If we need anything urgently in 2.13 then we can do a bug fix
 release 2.13.1.
 >
 >
 > On Tue, Jun 4, 2019 at 8:59 AM Thomas Weise 
 wrote:
 >>
 >> This seems a rushed and things fall through the cracks.
 >>
 >> Max had requested to not include the weekend into the voting
 period.
 >>
 >> Valentyn: I had the same question on the first RC. The PR should
 be included into the vote for review. You can find it here:
 https://github.com/apache/beam/pull/8667/files
 >>
 >> I had requested to include 

Re: Removing shading by default within BeamModulePlugin.groovy

[Lukasz Cwik]

I am able to pass several runners validates runner tests and the Java
PostCommit.

I also was able to publish a release into the staging repository[1] and
compared the newly generated poms artifact-2.14.0-20190605.*-30.pom against
the previously nightly snapshot of artifact-2.14.0-20190605.*-28.pom for
the following projects as a spot check and found no differences in those
poms:
beam-sdks-java-core
beam-sdks-java-fn-execution
beam-runners-spark

I believe my PR is now ready for review.

1: https://repository.apache.org/content/groups/snapshots/org/apache/beam/

On Tue, Jun 4, 2019 at 7:18 PM Kenneth Knowles  wrote:

> Nice! This is a huge step. One thing that showed up in the last big gradle
> change was needing to check the generated poms.
>
> Kenn
>
> On Tue, Jun 4, 2019 at 5:07 PM Lukasz Cwik  wrote:
>
>> Since we have been migrating to using vendoring instead of shading[1] and
>> due to previous efforts in vendoring[2, 3] I have opened up PR 8762[4]
>> which migrates all projects that weren't doing anything shading wise to not
>> perform any shading. This required me to fix up all intra project
>> dependencies and release publishing.
>>
>> The following is a list of all project paths which are still using
>> shading for some reason:
>> model/*
>> sdks/java/core
>> sdks/java/extensions/kryo
>> sdks/java/extensions/sql
>> sdks/java/extensions/sql/jdbc
>> sdks/java/harness
>> runners/spark/job-server
>> runners/direct-java
>> runners/samza/job-server
>> runners/google-cloud-dataflow-java/worker
>> runners/google-cloud-dataflow-java/worker/legacy-worker
>> runners/google-cloud-dataflow-java/worker/windmill
>> vendor/*
>>
>> Out of the list above, migrating sdks/java/core and runners/direct-java
>> (in that order) would provide the most benefit to moving away from shading
>> within our project. Many of the others are either shaded proto classes or
>> applications (e.g. job-servers, harness, sql jdbc) and either require
>> shading to be compatible with vendoring or aren't meant to be used as
>> dependencies.
>>
>> Since this is a larger change that cuts across so many projects there is
>> risk for breakage. I'm looking for people to help test the change and
>> validate any scenarios that they are specifically interested in. I'm
>> planning to run several of the postcommits on my PR and check that we can
>> build a release in addition to any efforts others provide before looking to
>> have the change merged.
>>
>> The following guidance should help those who edit Gradle build files
>> (after this change is merged):
>> * For projects that don't perform any shading, those projects have been
>> migrated to use the default configurations that the Gradle Java plugin
>> uses[5]. Note that the default configurations we use have been deprecated.
>> * For projects that depend on another project that isn't shaded, the
>> intra project configuration has been swapped to use compile / testRuntime
>> instead of shadow and shadowTest
>> * Existing projects that are still shaded should use the shadow and
>> shadowTest configurations as before.
>>
>> 1:
>> https://lists.apache.org/thread.html/4c12db35b40a6d56e170cd6fc8bb0ac4c43a99aa3cb7dbae54176815@%3Cdev.beam.apache.org%3E
>> 2:
>> https://lists.apache.org/thread.html/4c12db35b40a6d56e170cd6fc8bb0ac4c43a99aa3cb7dbae54176815@%3Cdev.beam.apache.org%3E
>> 3:
>> https://lists.apache.org/thread.html/972b5175641f4eaf7ec92870cc0ff72fa52e6f0bbaccc384a3814e45@%3Cdev.beam.apache.org%3E
>> 4: https://github.com/apache/beam/pull/8762
>> 5:
>> https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_plugin_and_dependency_management
>>
>

Re: [discuss] A tweak to the Python API for SDF?

[Pablo Estrada]

I have no objections.

+Ismaël Mejía  who has familiarity and interest in Java
SDF.

On Wed, Jun 5, 2019 at 11:31 AM Brian Hulette  wrote:

> Just wanted to resurrect this to say that it seems appropriate to make the
> same change in Java. All the same arguments apply there, and now there's
> the additional argument for maintaining symmetry with Python.
>
> I think BEAM-7250 should be changed to a ticket to actually implement this
> in Java unless someone has an objection.
>
> Brian
>
> On Wed, May 8, 2019 at 2:20 PM Pablo Estrada  wrote:
>
>> Hello all,
>> The API has been updated for Python (See
>> https://github.com/apache/beam/pull/8430). Please, if you catch any
>> documentation that needs updating, flag to me or just propose the change : )
>>
>> As for Java - we didn't end up determining whether it makes sense to
>> update the API as well. Thoughts from others?
>>
>> In any case, I've filed https://jira.apache.org/jira/browse/BEAM-7250 to
>> track this for Java.
>>
>> Best
>> -P.
>>
>> On Mon, Apr 29, 2019 at 2:41 PM Lukasz Cwik  wrote:
>>
>>> Pablo, all the SplittableDoFn stuff is marked as @Experimental so one is
>>> able to change it. There really is only one complicated one to change in
>>> Watch.java, the rest are quite straightforward.
>>>
>>> On Mon, Apr 29, 2019 at 2:23 PM Pablo Estrada 
>>> wrote:
>>>
 Thanks all,
  @Luke - I imagine that would be an improvement to the API, but this
 may be harder as this is already available to users, and there are those
 who have implemented SDFs under the current API. Would it be possible to
 make a backwards-compatible change to the API here?

 For the Python changes, I've proposed a pull request:
 https://github.com/apache/beam/pull/8430 - it was smaller than I
 thought : ) - All comments welcome please.

 +Boyuan Zhang  I am happy to wait for your
 SyntheticSource PR to be merged and make the appropriate changes if you'd
 like.
 Best
 -P.

 On Mon, Apr 29, 2019 at 8:23 AM Lukasz Cwik  wrote:

> Would it make sense to also do this in the Java SDK?
>
> The would make the restriction provider also mirror the TimerSpec and
> StateSpec which use annotations similar to how its done in Python.
>
> On Mon, Apr 29, 2019 at 3:42 AM Robert Bradshaw 
> wrote:
>
>> +1 to introducing this Param for consistency (and making the
>> substitution more obvious), and I think SDF is still new/experimental
>> enough we can do this. I don't know if we need Spec in addition to
>> Param and Provider.
>>
>> On Sat, Apr 27, 2019 at 1:07 AM Chamikara Jayalath <
>> chamik...@google.com> wrote:
>> >
>> >
>> >
>> > On Fri, Apr 26, 2019 at 3:43 PM Pablo Estrada 
>> wrote:
>> >>
>> >> Hi all,
>> >> Sorry about the wall of text.
>> >> So, first of all, I thought about this while reviewing a PR by
>> Boyuan with an example of an SDF[1]. This is very exciting btw : ).
>> >>
>> >> Anyway... I certainly have a limited view of the whole SDF effort,
>> but I think it's worth discussing this particular point about the API
>> before finalizing SDF and making it widely available. So here I go:
>> >>
>> >> The Python API for SDF asks users to provide a restriction
>> provider in their process function signature. More or less the following:
>> >>
>> >> class MyOwnLittleSDF(beam.DoFn):
>> >>   def process(self, element,
>> >>
>>  restriction_tracker=MyOwnLittleRestrictionProvider()):
>> >> # My DoFn logic...
>> >>
>> >> This is all fine, but something that I found a little odd is that
>> the restriction provider gets replaced at runtime with a restriction
>> tracker:
>> >>
>> >> class MyOwnLittleSDF(beam.DoFn):
>> >>   def process(self, element,
>> >>
>>  restriction_tracker=MyOwnLittleRestrictionProvider()):
>> >> # This assert succeeds : )
>> >> assert not isinstance(restriction_tracker,
>> >>   MyOwnLittleRestrictionProvider)
>> >>
>> >> After thinking a little bit about it, I realized that the default
>> argument simply allows us to inform the runner where to find the
>> restriction provider; but that the thing that we need at runtime is NOT 
>> the
>> restriction provider - but rather, the restriction tracker.
>> >>
>> >> A similar pattern occurs with state and timers, where the runner
>> needs to know the sort of state, the coder for the values in that state 
>> (or
>> the time domain for timers); but the runtime parameter is different[2]. 
>> For
>> state and timers (and window, timestamp, pane, etc.) we provide a pattern
>> where users give a default value that is clearly a placeholder:
>> beam.DoFn.TimerParam, or beam.DoFn.StateParam.
>> >
>> >
>> > This is the way (new) DoFn work for Python SDK. SDK (harness)

Re: [discuss] A tweak to the Python API for SDF?

[Brian Hulette]

Just wanted to resurrect this to say that it seems appropriate to make the
same change in Java. All the same arguments apply there, and now there's
the additional argument for maintaining symmetry with Python.

I think BEAM-7250 should be changed to a ticket to actually implement this
in Java unless someone has an objection.

Brian

On Wed, May 8, 2019 at 2:20 PM Pablo Estrada  wrote:

> Hello all,
> The API has been updated for Python (See
> https://github.com/apache/beam/pull/8430). Please, if you catch any
> documentation that needs updating, flag to me or just propose the change : )
>
> As for Java - we didn't end up determining whether it makes sense to
> update the API as well. Thoughts from others?
>
> In any case, I've filed https://jira.apache.org/jira/browse/BEAM-7250 to
> track this for Java.
>
> Best
> -P.
>
> On Mon, Apr 29, 2019 at 2:41 PM Lukasz Cwik  wrote:
>
>> Pablo, all the SplittableDoFn stuff is marked as @Experimental so one is
>> able to change it. There really is only one complicated one to change in
>> Watch.java, the rest are quite straightforward.
>>
>> On Mon, Apr 29, 2019 at 2:23 PM Pablo Estrada  wrote:
>>
>>> Thanks all,
>>>  @Luke - I imagine that would be an improvement to the API, but this may
>>> be harder as this is already available to users, and there are those who
>>> have implemented SDFs under the current API. Would it be possible to make a
>>> backwards-compatible change to the API here?
>>>
>>> For the Python changes, I've proposed a pull request:
>>> https://github.com/apache/beam/pull/8430 - it was smaller than I
>>> thought : ) - All comments welcome please.
>>>
>>> +Boyuan Zhang  I am happy to wait for your
>>> SyntheticSource PR to be merged and make the appropriate changes if you'd
>>> like.
>>> Best
>>> -P.
>>>
>>> On Mon, Apr 29, 2019 at 8:23 AM Lukasz Cwik  wrote:
>>>
 Would it make sense to also do this in the Java SDK?

 The would make the restriction provider also mirror the TimerSpec and
 StateSpec which use annotations similar to how its done in Python.

 On Mon, Apr 29, 2019 at 3:42 AM Robert Bradshaw 
 wrote:

> +1 to introducing this Param for consistency (and making the
> substitution more obvious), and I think SDF is still new/experimental
> enough we can do this. I don't know if we need Spec in addition to
> Param and Provider.
>
> On Sat, Apr 27, 2019 at 1:07 AM Chamikara Jayalath <
> chamik...@google.com> wrote:
> >
> >
> >
> > On Fri, Apr 26, 2019 at 3:43 PM Pablo Estrada 
> wrote:
> >>
> >> Hi all,
> >> Sorry about the wall of text.
> >> So, first of all, I thought about this while reviewing a PR by
> Boyuan with an example of an SDF[1]. This is very exciting btw : ).
> >>
> >> Anyway... I certainly have a limited view of the whole SDF effort,
> but I think it's worth discussing this particular point about the API
> before finalizing SDF and making it widely available. So here I go:
> >>
> >> The Python API for SDF asks users to provide a restriction provider
> in their process function signature. More or less the following:
> >>
> >> class MyOwnLittleSDF(beam.DoFn):
> >>   def process(self, element,
> >>   restriction_tracker=MyOwnLittleRestrictionProvider()):
> >> # My DoFn logic...
> >>
> >> This is all fine, but something that I found a little odd is that
> the restriction provider gets replaced at runtime with a restriction
> tracker:
> >>
> >> class MyOwnLittleSDF(beam.DoFn):
> >>   def process(self, element,
> >>   restriction_tracker=MyOwnLittleRestrictionProvider()):
> >> # This assert succeeds : )
> >> assert not isinstance(restriction_tracker,
> >>   MyOwnLittleRestrictionProvider)
> >>
> >> After thinking a little bit about it, I realized that the default
> argument simply allows us to inform the runner where to find the
> restriction provider; but that the thing that we need at runtime is NOT 
> the
> restriction provider - but rather, the restriction tracker.
> >>
> >> A similar pattern occurs with state and timers, where the runner
> needs to know the sort of state, the coder for the values in that state 
> (or
> the time domain for timers); but the runtime parameter is different[2]. 
> For
> state and timers (and window, timestamp, pane, etc.) we provide a pattern
> where users give a default value that is clearly a placeholder:
> beam.DoFn.TimerParam, or beam.DoFn.StateParam.
> >
> >
> > This is the way (new) DoFn work for Python SDK. SDK (harness)
> identifies meanings of different (potential) arguments to a DoFn using
> pre-defined default values.
> >
> >>
> >>
> >> In this case, the API is fairly similar, but (at least in my
> imagination), it is much more clear about how 

Re: [VOTE] Release 2.13.0, release candidate #2

[Kenneth Knowles]

Just discovered a potentially serious issue that was present during this
RC: https://issues.apache.org/jira/browse/BEAM-7493. So far I have not
discovered a truly user-facing impact, and example validation succeeded,
but I wanted to alert the list.

Summary: When rendering a published pom.xml the dependencies are always the
path concatenated with dashes even when that is not the correct artifactId.
For example sdks/java/testing/test-utils are resolved in the pom to
beam-sdks-java-testing-test-utils. This does not exist; it manually sets
the name to beam-sdks-java-test-utils, omitting the extraneous `testing`
directory that exists only for taxonomy.

There are a few other modules that manually set their archive name. From
what I can tell, each of these is either (a) not published or (b) not
depended upon. I am still checking.

Kenn


On Wed, Jun 5, 2019 at 9:22 AM Thomas Weise  wrote:

> +1 and I think all of that can be covered with JIRA.
>
> Irrespective the release manager still needs to pay attention to the
> communication on the VOTE thread.
>
> On Wed, Jun 5, 2019 at 9:19 AM Ahmet Altay  wrote:
>
>> Checking that JIRA link sounds reasonable as long as we can agree that it
>> is single source of truth for cherry pick requests. I also agree with Cham,
>> requests need to come with a reason.
>>
>> On Wed, Jun 5, 2019 at 7:38 AM Ismaël Mejía  wrote:
>>
>>> I don't think we need anything fancier or marking even as Blocker some
>>> of this stuff, would not be enough just to monitor that [1] has no
>>> issues? (of course if the interested party has not put the fix version
>>> to the current ongoing vote one this is a mistake).
>>>
>>> [1]
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.13.0
>>>
>>> On Wed, Jun 5, 2019 at 4:23 PM Chamikara Jayalath 
>>> wrote:
>>> >
>>> >
>>> >
>>> > On Tue, Jun 4, 2019 at 5:02 PM Ahmet Altay  wrote:
>>> >>
>>> >> I would suggest have a single way of tracking cherry pick request to
>>> an RC. Currently we use emails on the RC thread, open PRs, and Jiras tagged
>>> for the release. This is confusing the person doing the release while they
>>> are juggling multiple things. How about we ask all cherry pick requests to
>>> have a JIRA filed against that release and marked as blockers?
>>> >
>>> >
>>> > I agree with this and with what Ankur said. Release blockers should be
>>> more explicit and should come with a reason. After voting thread start, I
>>> would say this should include a mail to the voting thread as well as a
>>> blocking JIRA. Other PRs opened against the branch may or may not get
>>> merged at the discretion of the release manager.
>>> >
>>> > Thanks,
>>> > Cham
>>> >
>>> >>
>>> >>
>>> >> On Tue, Jun 4, 2019 at 1:05 PM Ankur Goenka 
>>> wrote:
>>> >>>
>>> >>> That makes sense.
>>> >>> I would also like to add that the corresponding PR should be added
>>> to an open blocking Jira for the release to keep a single source to check.
>>> >>>
>>> >>> On Tue, Jun 4, 2019 at 12:15 PM Kenneth Knowles 
>>> wrote:
>>> 
>>>  I would actually suggest that the following search needs to be
>>> triaged to zero before cutting an RC:
>>> https://github.com/apache/beam/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+base%3Arelease-2.13.0
>>> .
>>> 
>>>  On Tue, Jun 4, 2019 at 11:17 AM Ankur Goenka 
>>> wrote:
>>> >
>>> > Sorry, I missed the comment for not including weekend's to 72
>>> hours voting period.
>>> >
>>> > I meant to update the blog post
>>> https://github.com/apache/beam/pull/8667/files once we have finalized
>>> the RC so that it can be consistent. Please add any comments to PR and I
>>> can incorporate them.
>>> >
>>> > As we did not go for 3rd RC and
>>> https://github.com/apache/beam/pull/8714 was not blocking the 2.13
>>> release, I went with the release.
>>> >
>>> > I have released the maven artifacts for beam. So I suppose, we can
>>> not do another RC for 2.13.0.
>>> > If we need anything urgently in 2.13 then we can do a bug fix
>>> release 2.13.1.
>>> >
>>> >
>>> > On Tue, Jun 4, 2019 at 8:59 AM Thomas Weise 
>>> wrote:
>>> >>
>>> >> This seems a rushed and things fall through the cracks.
>>> >>
>>> >> Max had requested to not include the weekend into the voting
>>> period.
>>> >>
>>> >> Valentyn: I had the same question on the first RC. The PR should
>>> be included into the vote for review. You can find it here:
>>> https://github.com/apache/beam/pull/8667/files
>>> >>
>>> >> I had requested to include following backport PR before the RC:
>>> https://github.com/apache/beam/pull/8714  - It's not blocking but would
>>> be nice if someone can merge it for any future release from this branch.
>>> >>
>>> >> Thanks,
>>> >> Thomas
>>> >>
>>> >>
>>> >> On Tue, Jun 4, 2019 at 1:59 AM Maximilian Michels 
>>> wrote:
>>> >>>
>>> >>> The 

Re: Help triaging Jira issues

[Tanay Tummalapalli]

Hi Kenneth,

I already follow the issues@ mailing list pretty much daily.
I'd like to help with triaging issues, especially ones related to the
Python SDK since I'm most familiar with it.

On Wed, Jun 5, 2019 at 10:26 PM Alex Van Boxel  wrote:

> Hey Kenneth, I help out. I'm planning to contribute more on Beam and it
> seems to be ideal to keep up-to-date with the project.
>
>  _/
> _/ Alex Van Boxel
>
>
> On Wed, Jun 5, 2019 at 6:46 PM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> I am requesting help in triaging incoming issues. I made a search here:
>> https://issues.apache.org/jira/issues/?filter=12345682
>> 
>>
>> I have a daily email subscription to this filter as a reminder, but
>> rarely can really sit down to do triage for very long. It has grown from
>> just under 200 to just over 200. The rate is actually pretty low but there
>> is a backlog. I also want to start re-triaging stale bugs but priority
>> would be (1) keep up with new bugs (2) clear backlog (3) re-triage stale
>> bugs.
>>
>> Just FYI what I look for before I clicked "Triaged" is:
>>
>>  - correct component
>>  - correct priority
>>  - maybe ping someone in a comment or assign
>>  - write to dev@ if it is a major problem
>>
>> If I can't figure that out, then I ask the reporter for clarification and
>> "Start Watching" the issue so I will receive their response.
>>
>> To avoid duplicate triage work it may help to assign to yourself
>> temporarily during triage phase.
>>
>> Any help greatly appreciated!
>>
>> Kenn
>>
>

Re: Help triaging Jira issues

[Alex Van Boxel]

Hey Kenneth, I help out. I'm planning to contribute more on Beam and it
seems to be ideal to keep up-to-date with the project.

 _/
_/ Alex Van Boxel


On Wed, Jun 5, 2019 at 6:46 PM Kenneth Knowles  wrote:

> Hi all,
>
> I am requesting help in triaging incoming issues. I made a search here:
> https://issues.apache.org/jira/issues/?filter=12345682
> 
>
> I have a daily email subscription to this filter as a reminder, but rarely
> can really sit down to do triage for very long. It has grown from just
> under 200 to just over 200. The rate is actually pretty low but there is a
> backlog. I also want to start re-triaging stale bugs but priority would be
> (1) keep up with new bugs (2) clear backlog (3) re-triage stale bugs.
>
> Just FYI what I look for before I clicked "Triaged" is:
>
>  - correct component
>  - correct priority
>  - maybe ping someone in a comment or assign
>  - write to dev@ if it is a major problem
>
> If I can't figure that out, then I ask the reporter for clarification and
> "Start Watching" the issue so I will receive their response.
>
> To avoid duplicate triage work it may help to assign to yourself
> temporarily during triage phase.
>
> Any help greatly appreciated!
>
> Kenn
>

Help triaging Jira issues

[Kenneth Knowles]

Hi all,

I am requesting help in triaging incoming issues. I made a search here:
https://issues.apache.org/jira/issues/?filter=12345682


I have a daily email subscription to this filter as a reminder, but rarely
can really sit down to do triage for very long. It has grown from just
under 200 to just over 200. The rate is actually pretty low but there is a
backlog. I also want to start re-triaging stale bugs but priority would be
(1) keep up with new bugs (2) clear backlog (3) re-triage stale bugs.

Just FYI what I look for before I clicked "Triaged" is:

 - correct component
 - correct priority
 - maybe ping someone in a comment or assign
 - write to dev@ if it is a major problem

If I can't figure that out, then I ask the reporter for clarification and
"Start Watching" the issue so I will receive their response.

To avoid duplicate triage work it may help to assign to yourself
temporarily during triage phase.

Any help greatly appreciated!

Kenn

Re: Plan for dropping python 2 support

[Ahmet Altay]

I agree with the sentiment on this thread. Our priority needs to be
offering good python 3 support that we can comfortably recommend users to
switch. Progress on that so far has been promising and I do anticipate that
we will reach there in the near future.

My proposal would be, once we reach to that state, we can mark the first
subsequent Beam release as the last Beam release that supports Python 2.
(Alternatively: in line with the previous experimental/deprecated
discussion we can make 2 more release with python 2 support rather than
just 1 more.) With the current state, we would not give users plenty of
time to upgrade python 3. So in addition, I would suggest we can consider
and upgrade relief by offering something like a 6-month support on the last
python 2 compatible release. We might do that in the context of an LTS
release.

I do not believe we have a timeline we can share with users at this point.
However if we go with this suggestion, we will probably support python 2
approximately until mid-2020.

Ahmet

On Wed, Jun 5, 2019 at 4:53 AM Tanay Tummalapalli 
wrote:

> We can support Python 2 for some time in 2020, but, we should target a
> date no later than 2020 to drop support.
> If we do plan to drop support for Python 2 in 2020, we should sign the
> Python 3 statement[1], declaring that we will "drop support for Python 2.7
> no later than 2020".
>
> In addition to the statement, keeping a target release and date(if
> possible) or timeline to drop support would also help users to decide when
> they need to work on migrating to Python 3.
>
> Regards,
> - TT
>
> [1] https://python3statement.org/
>
> On Wed, Jun 5, 2019 at 4:37 PM Robert Bradshaw 
> wrote:
>
>> Until Python 3 support for Beam is officially out of beta and
>> recommended, I don't think we can tell people to stop using Python 2.
>> Given that 2020 is just over 6 months away, that seems a short
>> transition time, so I would guess we'll have to continue supporting
>> Python 2 sometime into 2020.
>>
>> A quick survey of users would be valuable here. But first priority is
>> making Python 3 rock solid so we can unconditionally recommend it over
>> Python 2.
>>
>> On Wed, Jun 5, 2019 at 12:27 PM Ismaël Mejía  wrote:
>> >
>> > Python 2 won't be maintained after 2020 [1]. I was wondering what will
>> > be our (Beam) plan for this. Other projects [2] have started to alert
>> > users that support will be removed so maybe we should decide or policy
>> > for this too.
>> >
>> > [1] https://pythonclock.org/
>> > [2]
>> https://spark.apache.org/news/plan-for-dropping-python-2-support.html
>>
>

Re: [VOTE] Release 2.13.0, release candidate #2

[Thomas Weise]

+1 and I think all of that can be covered with JIRA.

Irrespective the release manager still needs to pay attention to the
communication on the VOTE thread.

On Wed, Jun 5, 2019 at 9:19 AM Ahmet Altay  wrote:

> Checking that JIRA link sounds reasonable as long as we can agree that it
> is single source of truth for cherry pick requests. I also agree with Cham,
> requests need to come with a reason.
>
> On Wed, Jun 5, 2019 at 7:38 AM Ismaël Mejía  wrote:
>
>> I don't think we need anything fancier or marking even as Blocker some
>> of this stuff, would not be enough just to monitor that [1] has no
>> issues? (of course if the interested party has not put the fix version
>> to the current ongoing vote one this is a mistake).
>>
>> [1]
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.13.0
>>
>> On Wed, Jun 5, 2019 at 4:23 PM Chamikara Jayalath 
>> wrote:
>> >
>> >
>> >
>> > On Tue, Jun 4, 2019 at 5:02 PM Ahmet Altay  wrote:
>> >>
>> >> I would suggest have a single way of tracking cherry pick request to
>> an RC. Currently we use emails on the RC thread, open PRs, and Jiras tagged
>> for the release. This is confusing the person doing the release while they
>> are juggling multiple things. How about we ask all cherry pick requests to
>> have a JIRA filed against that release and marked as blockers?
>> >
>> >
>> > I agree with this and with what Ankur said. Release blockers should be
>> more explicit and should come with a reason. After voting thread start, I
>> would say this should include a mail to the voting thread as well as a
>> blocking JIRA. Other PRs opened against the branch may or may not get
>> merged at the discretion of the release manager.
>> >
>> > Thanks,
>> > Cham
>> >
>> >>
>> >>
>> >> On Tue, Jun 4, 2019 at 1:05 PM Ankur Goenka  wrote:
>> >>>
>> >>> That makes sense.
>> >>> I would also like to add that the corresponding PR should be added to
>> an open blocking Jira for the release to keep a single source to check.
>> >>>
>> >>> On Tue, Jun 4, 2019 at 12:15 PM Kenneth Knowles 
>> wrote:
>> 
>>  I would actually suggest that the following search needs to be
>> triaged to zero before cutting an RC:
>> https://github.com/apache/beam/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+base%3Arelease-2.13.0
>> .
>> 
>>  On Tue, Jun 4, 2019 at 11:17 AM Ankur Goenka 
>> wrote:
>> >
>> > Sorry, I missed the comment for not including weekend's to 72 hours
>> voting period.
>> >
>> > I meant to update the blog post
>> https://github.com/apache/beam/pull/8667/files once we have finalized
>> the RC so that it can be consistent. Please add any comments to PR and I
>> can incorporate them.
>> >
>> > As we did not go for 3rd RC and
>> https://github.com/apache/beam/pull/8714 was not blocking the 2.13
>> release, I went with the release.
>> >
>> > I have released the maven artifacts for beam. So I suppose, we can
>> not do another RC for 2.13.0.
>> > If we need anything urgently in 2.13 then we can do a bug fix
>> release 2.13.1.
>> >
>> >
>> > On Tue, Jun 4, 2019 at 8:59 AM Thomas Weise  wrote:
>> >>
>> >> This seems a rushed and things fall through the cracks.
>> >>
>> >> Max had requested to not include the weekend into the voting
>> period.
>> >>
>> >> Valentyn: I had the same question on the first RC. The PR should
>> be included into the vote for review. You can find it here:
>> https://github.com/apache/beam/pull/8667/files
>> >>
>> >> I had requested to include following backport PR before the RC:
>> https://github.com/apache/beam/pull/8714  - It's not blocking but would
>> be nice if someone can merge it for any future release from this branch.
>> >>
>> >> Thanks,
>> >> Thomas
>> >>
>> >>
>> >> On Tue, Jun 4, 2019 at 1:59 AM Maximilian Michels 
>> wrote:
>> >>>
>> >>> The summary is not correct. Binding votes (in order):
>> >>>
>> >>> Ahmet Altay
>> >>> Robert Bradshaw
>> >>> Maximilian Michels
>> >>> Jean-Baptiste Onofré
>> >>> Lukasz Cwik
>> >>>
>> >>> A total of 5 binding votes.
>> >>>
>> >>> On 04.06.19 02:37, Ankur Goenka wrote:
>> >>> > +1
>> >>> > Thanks for validating the release and voting.
>> >>> > With 0(-1), 6(+1) and 3(+1 binding) votes, I am concluding the
>> voting
>> >>> > process.
>> >>> > I am going ahead with the release and will keep the community
>> posted
>> >>> > with the updates.
>> >>> >
>> >>> > On Mon, Jun 3, 2019 at 1:57 PM Andrew Pilloud <
>> apill...@google.com
>> >>> > > wrote:
>> >>> >
>> >>> > +1 Reviewed the Nexmark java and SQL perfkit graphs, no
>> obvious
>> >>> > regressions over the previous release.
>> >>> >
>> >>> > On Mon, Jun 3, 2019 at 1:15 PM Lukasz Cwik <
>> lc...@google.com
>> >>> >

Re: [VOTE] Release 2.13.0, release candidate #2

[Ahmet Altay]

Checking that JIRA link sounds reasonable as long as we can agree that it
is single source of truth for cherry pick requests. I also agree with Cham,
requests need to come with a reason.

On Wed, Jun 5, 2019 at 7:38 AM Ismaël Mejía  wrote:

> I don't think we need anything fancier or marking even as Blocker some
> of this stuff, would not be enough just to monitor that [1] has no
> issues? (of course if the interested party has not put the fix version
> to the current ongoing vote one this is a mistake).
>
> [1]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.13.0
>
> On Wed, Jun 5, 2019 at 4:23 PM Chamikara Jayalath 
> wrote:
> >
> >
> >
> > On Tue, Jun 4, 2019 at 5:02 PM Ahmet Altay  wrote:
> >>
> >> I would suggest have a single way of tracking cherry pick request to an
> RC. Currently we use emails on the RC thread, open PRs, and Jiras tagged
> for the release. This is confusing the person doing the release while they
> are juggling multiple things. How about we ask all cherry pick requests to
> have a JIRA filed against that release and marked as blockers?
> >
> >
> > I agree with this and with what Ankur said. Release blockers should be
> more explicit and should come with a reason. After voting thread start, I
> would say this should include a mail to the voting thread as well as a
> blocking JIRA. Other PRs opened against the branch may or may not get
> merged at the discretion of the release manager.
> >
> > Thanks,
> > Cham
> >
> >>
> >>
> >> On Tue, Jun 4, 2019 at 1:05 PM Ankur Goenka  wrote:
> >>>
> >>> That makes sense.
> >>> I would also like to add that the corresponding PR should be added to
> an open blocking Jira for the release to keep a single source to check.
> >>>
> >>> On Tue, Jun 4, 2019 at 12:15 PM Kenneth Knowles 
> wrote:
> 
>  I would actually suggest that the following search needs to be
> triaged to zero before cutting an RC:
> https://github.com/apache/beam/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+base%3Arelease-2.13.0
> .
> 
>  On Tue, Jun 4, 2019 at 11:17 AM Ankur Goenka 
> wrote:
> >
> > Sorry, I missed the comment for not including weekend's to 72 hours
> voting period.
> >
> > I meant to update the blog post
> https://github.com/apache/beam/pull/8667/files once we have finalized the
> RC so that it can be consistent. Please add any comments to PR and I can
> incorporate them.
> >
> > As we did not go for 3rd RC and
> https://github.com/apache/beam/pull/8714 was not blocking the 2.13
> release, I went with the release.
> >
> > I have released the maven artifacts for beam. So I suppose, we can
> not do another RC for 2.13.0.
> > If we need anything urgently in 2.13 then we can do a bug fix
> release 2.13.1.
> >
> >
> > On Tue, Jun 4, 2019 at 8:59 AM Thomas Weise  wrote:
> >>
> >> This seems a rushed and things fall through the cracks.
> >>
> >> Max had requested to not include the weekend into the voting period.
> >>
> >> Valentyn: I had the same question on the first RC. The PR should be
> included into the vote for review. You can find it here:
> https://github.com/apache/beam/pull/8667/files
> >>
> >> I had requested to include following backport PR before the RC:
> https://github.com/apache/beam/pull/8714  - It's not blocking but would
> be nice if someone can merge it for any future release from this branch.
> >>
> >> Thanks,
> >> Thomas
> >>
> >>
> >> On Tue, Jun 4, 2019 at 1:59 AM Maximilian Michels 
> wrote:
> >>>
> >>> The summary is not correct. Binding votes (in order):
> >>>
> >>> Ahmet Altay
> >>> Robert Bradshaw
> >>> Maximilian Michels
> >>> Jean-Baptiste Onofré
> >>> Lukasz Cwik
> >>>
> >>> A total of 5 binding votes.
> >>>
> >>> On 04.06.19 02:37, Ankur Goenka wrote:
> >>> > +1
> >>> > Thanks for validating the release and voting.
> >>> > With 0(-1), 6(+1) and 3(+1 binding) votes, I am concluding the
> voting
> >>> > process.
> >>> > I am going ahead with the release and will keep the community
> posted
> >>> > with the updates.
> >>> >
> >>> > On Mon, Jun 3, 2019 at 1:57 PM Andrew Pilloud <
> apill...@google.com
> >>> > > wrote:
> >>> >
> >>> > +1 Reviewed the Nexmark java and SQL perfkit graphs, no
> obvious
> >>> > regressions over the previous release.
> >>> >
> >>> > On Mon, Jun 3, 2019 at 1:15 PM Lukasz Cwik  >>> > > wrote:
> >>> >
> >>> > Thanks for the clarification.
> >>> >
> >>> > On Mon, Jun 3, 2019 at 11:40 AM Ankur Goenka <
> goe...@google.com
> >>> > > wrote:
> >>> >
> >>> > Yes, i meant i will close the voting at 5pm and
> start the
> >>> >   

Re: [VOTE] Release 2.13.0, release candidate #2

[Ismaël Mejía]

I don't think we need anything fancier or marking even as Blocker some
of this stuff, would not be enough just to monitor that [1] has no
issues? (of course if the interested party has not put the fix version
to the current ongoing vote one this is a mistake).

[1] 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.13.0

On Wed, Jun 5, 2019 at 4:23 PM Chamikara Jayalath  wrote:
>
>
>
> On Tue, Jun 4, 2019 at 5:02 PM Ahmet Altay  wrote:
>>
>> I would suggest have a single way of tracking cherry pick request to an RC. 
>> Currently we use emails on the RC thread, open PRs, and Jiras tagged for the 
>> release. This is confusing the person doing the release while they are 
>> juggling multiple things. How about we ask all cherry pick requests to have 
>> a JIRA filed against that release and marked as blockers?
>
>
> I agree with this and with what Ankur said. Release blockers should be more 
> explicit and should come with a reason. After voting thread start, I would 
> say this should include a mail to the voting thread as well as a blocking 
> JIRA. Other PRs opened against the branch may or may not get merged at the 
> discretion of the release manager.
>
> Thanks,
> Cham
>
>>
>>
>> On Tue, Jun 4, 2019 at 1:05 PM Ankur Goenka  wrote:
>>>
>>> That makes sense.
>>> I would also like to add that the corresponding PR should be added to an 
>>> open blocking Jira for the release to keep a single source to check.
>>>
>>> On Tue, Jun 4, 2019 at 12:15 PM Kenneth Knowles  wrote:

 I would actually suggest that the following search needs to be triaged to 
 zero before cutting an RC: 
 https://github.com/apache/beam/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+base%3Arelease-2.13.0.

 On Tue, Jun 4, 2019 at 11:17 AM Ankur Goenka  wrote:
>
> Sorry, I missed the comment for not including weekend's to 72 hours 
> voting period.
>
> I meant to update the blog post 
> https://github.com/apache/beam/pull/8667/files once we have finalized the 
> RC so that it can be consistent. Please add any comments to PR and I can 
> incorporate them.
>
> As we did not go for 3rd RC and https://github.com/apache/beam/pull/8714 
> was not blocking the 2.13 release, I went with the release.
>
> I have released the maven artifacts for beam. So I suppose, we can not do 
> another RC for 2.13.0.
> If we need anything urgently in 2.13 then we can do a bug fix release 
> 2.13.1.
>
>
> On Tue, Jun 4, 2019 at 8:59 AM Thomas Weise  wrote:
>>
>> This seems a rushed and things fall through the cracks.
>>
>> Max had requested to not include the weekend into the voting period.
>>
>> Valentyn: I had the same question on the first RC. The PR should be 
>> included into the vote for review. You can find it here: 
>> https://github.com/apache/beam/pull/8667/files
>>
>> I had requested to include following backport PR before the RC: 
>> https://github.com/apache/beam/pull/8714  - It's not blocking but would 
>> be nice if someone can merge it for any future release from this branch.
>>
>> Thanks,
>> Thomas
>>
>>
>> On Tue, Jun 4, 2019 at 1:59 AM Maximilian Michels  
>> wrote:
>>>
>>> The summary is not correct. Binding votes (in order):
>>>
>>> Ahmet Altay
>>> Robert Bradshaw
>>> Maximilian Michels
>>> Jean-Baptiste Onofré
>>> Lukasz Cwik
>>>
>>> A total of 5 binding votes.
>>>
>>> On 04.06.19 02:37, Ankur Goenka wrote:
>>> > +1
>>> > Thanks for validating the release and voting.
>>> > With 0(-1), 6(+1) and 3(+1 binding) votes, I am concluding the voting
>>> > process.
>>> > I am going ahead with the release and will keep the community posted
>>> > with the updates.
>>> >
>>> > On Mon, Jun 3, 2019 at 1:57 PM Andrew Pilloud >> > > wrote:
>>> >
>>> > +1 Reviewed the Nexmark java and SQL perfkit graphs, no obvious
>>> > regressions over the previous release.
>>> >
>>> > On Mon, Jun 3, 2019 at 1:15 PM Lukasz Cwik >> > > wrote:
>>> >
>>> > Thanks for the clarification.
>>> >
>>> > On Mon, Jun 3, 2019 at 11:40 AM Ankur Goenka 
>>> > >> > > wrote:
>>> >
>>> > Yes, i meant i will close the voting at 5pm and start the
>>> > release process.
>>> >
>>> > On Mon, Jun 3, 2019, 10:59 AM Lukasz Cwik 
>>> > >> > > wrote:
>>> >
>>> > Ankur, did you mean to say your going to close the 
>>> > vote
>>> > today at 5pm? (and then complete the release 
>>> > afterwards)
>>> >
>>> > 

Re: [VOTE] Release 2.13.0, release candidate #2

[Chamikara Jayalath]

On Tue, Jun 4, 2019 at 5:02 PM Ahmet Altay  wrote:

> I would suggest have a single way of tracking cherry pick request to an
> RC. Currently we use emails on the RC thread, open PRs, and Jiras tagged
> for the release. This is confusing the person doing the release while they
> are juggling multiple things. How about we ask all cherry pick requests to
> have a JIRA filed against that release and marked as blockers?
>

I agree with this and with what Ankur said. Release blockers should be more
explicit and should come with a reason. After voting thread start, I would
say this should include a mail to the voting thread as well as a blocking
JIRA. Other PRs opened against the branch may or may not get merged at the
discretion of the release manager.

Thanks,
Cham


>
> On Tue, Jun 4, 2019 at 1:05 PM Ankur Goenka  wrote:
>
>> That makes sense.
>> I would also like to add that the corresponding PR should be added to an
>> open blocking Jira
>>  for the
>> release to keep a single source to check.
>>
>> On Tue, Jun 4, 2019 at 12:15 PM Kenneth Knowles  wrote:
>>
>>> I would actually suggest that the following search needs to be triaged
>>> to zero before cutting an RC:
>>> https://github.com/apache/beam/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+base%3Arelease-2.13.0
>>> .
>>>
>>> On Tue, Jun 4, 2019 at 11:17 AM Ankur Goenka  wrote:
>>>
 Sorry, I missed the comment for not including weekend's to 72 hours
 voting period.

 I meant to update the blog post
 https://github.com/apache/beam/pull/8667/files once we have finalized
 the RC so that it can be consistent. Please add any comments to PR and I
 can incorporate them.

 As we did not go for 3rd RC and
 https://github.com/apache/beam/pull/8714 was not blocking the 2.13
 release, I went with the release.

 I have released the maven artifacts for beam. So I suppose, we can not
 do another RC for 2.13.0.
 If we need anything urgently in 2.13 then we can do a bug fix release
 2.13.1.


 On Tue, Jun 4, 2019 at 8:59 AM Thomas Weise  wrote:

> This seems a rushed and things fall through the cracks.
>
> Max had requested to not include the weekend into the voting period.
>
> Valentyn: I had the same question on the first RC. The PR should be
> included into the vote for review. You can find it here:
> https://github.com/apache/beam/pull/8667/files
>
> I had requested to include following backport PR before the RC:
> https://github.com/apache/beam/pull/8714  - It's not blocking but
> would be nice if someone can merge it for any future release from this
> branch.
>
> Thanks,
> Thomas
>
>
> On Tue, Jun 4, 2019 at 1:59 AM Maximilian Michels 
> wrote:
>
>> The summary is not correct. Binding votes (in order):
>>
>> Ahmet Altay
>> Robert Bradshaw
>> Maximilian Michels
>> Jean-Baptiste Onofré
>> Lukasz Cwik
>>
>> A total of 5 binding votes.
>>
>> On 04.06.19 02:37, Ankur Goenka wrote:
>> > +1
>> > Thanks for validating the release and voting.
>> > With 0(-1), 6(+1) and 3(+1 binding) votes, I am concluding the
>> voting
>> > process.
>> > I am going ahead with the release and will keep the community
>> posted
>> > with the updates.
>> >
>> > On Mon, Jun 3, 2019 at 1:57 PM Andrew Pilloud > > > wrote:
>> >
>> > +1 Reviewed the Nexmark java and SQL perfkit graphs, no obvious
>> > regressions over the previous release.
>> >
>> > On Mon, Jun 3, 2019 at 1:15 PM Lukasz Cwik > > > wrote:
>> >
>> > Thanks for the clarification.
>> >
>> > On Mon, Jun 3, 2019 at 11:40 AM Ankur Goenka <
>> goe...@google.com
>> > > wrote:
>> >
>> > Yes, i meant i will close the voting at 5pm and start
>> the
>> > release process.
>> >
>> > On Mon, Jun 3, 2019, 10:59 AM Lukasz Cwik <
>> lc...@google.com
>> > > wrote:
>> >
>> > Ankur, did you mean to say your going to close the
>> vote
>> > today at 5pm? (and then complete the release
>> afterwards)
>> >
>> > On Mon, Jun 3, 2019 at 10:54 AM Ankur Goenka
>> > mailto:goe...@google.com>>
>> wrote:
>> >
>> > Thanks for validating and voting.
>> >
>> > We have 4 binding votes.
>> > I will complete the release today 5PM. Please
>> raise
>> > any concerns before that.
>> >
>> > Thanks,
>> > Ankur
>> >
>> > On Mon, 

Re: Plan for dropping python 2 support

[Tanay Tummalapalli]

We can support Python 2 for some time in 2020, but, we should target a date
no later than 2020 to drop support.
If we do plan to drop support for Python 2 in 2020, we should sign the
Python 3 statement[1], declaring that we will "drop support for Python 2.7
no later than 2020".

In addition to the statement, keeping a target release and date(if
possible) or timeline to drop support would also help users to decide when
they need to work on migrating to Python 3.

Regards,
- TT

[1] https://python3statement.org/

On Wed, Jun 5, 2019 at 4:37 PM Robert Bradshaw  wrote:

> Until Python 3 support for Beam is officially out of beta and
> recommended, I don't think we can tell people to stop using Python 2.
> Given that 2020 is just over 6 months away, that seems a short
> transition time, so I would guess we'll have to continue supporting
> Python 2 sometime into 2020.
>
> A quick survey of users would be valuable here. But first priority is
> making Python 3 rock solid so we can unconditionally recommend it over
> Python 2.
>
> On Wed, Jun 5, 2019 at 12:27 PM Ismaël Mejía  wrote:
> >
> > Python 2 won't be maintained after 2020 [1]. I was wondering what will
> > be our (Beam) plan for this. Other projects [2] have started to alert
> > users that support will be removed so maybe we should decide or policy
> > for this too.
> >
> > [1] https://pythonclock.org/
> > [2]
> https://spark.apache.org/news/plan-for-dropping-python-2-support.html
>

Re: [Discuss] Ideas for Apache Beam presence in social media

[Thomas Weise]

+1

What would be the mechanism to notify the PMC that there is something to
review?


On Tue, Jun 4, 2019 at 9:55 PM Kenneth Knowles  wrote:

> Bringing the PMC's conclusion back to this list, we are happy to start
> with the following arrangement:
>
>  - Doc/spreadsheet/etc readable by dev@ (aka the public), writable by
> some group of contributors to set up a queue of news
>  - Any member of PMC approves and executes the posts, with enough time
> elapsing to consider it lazy consensus
>
> Any mistake transcribing this conclusion is my own. And of course nothing
> is permanent, but we try and iterate.
>
> Kenn
>
> On Mon, Jun 3, 2019 at 2:18 PM Aizhamal Nurmamat kyzy 
> wrote:
>
>> Hello folks,
>>
>> I have created a spreadsheet where people can suggest tweets [1]. It
>> contains a couple of tweets that have been tweeted as examples. Also, there
>> are a couple others that I will ask PMC members to review in the next few
>> days.
>>
>> I have also created a blog post[2] to invite community members to
>> participate by proposing tweets / retweets.
>>
>> Does this look OK to everyone? I’d love to try it out and see if it
>> drives engagement in the community. If not we can always change the
>> processes.
>>
>> Thanks,
>> aizhamal
>>
>> [1] s.apache.org/beam-tweets
>> [2] https://github.com/apache/beam/pull/8747
>>
>> On Fri, May 24, 2019 at 4:26 PM Kenneth Knowles  wrote:
>>
>>> Thanks for taking on this work!
>>>
>>> Kenn
>>>
>>> On Fri, May 24, 2019 at 2:52 PM Aizhamal Nurmamat kyzy <
>>> aizha...@google.com> wrote:
>>>
 Hi everyone,

 I'd like to pilot this if that's okay by everyone. I'll set up a
 spreadsheet, write a blog post publicizing it, and perhaps send out a
 tweet. We can improve the process later with tools if necessary.

 Thanks all and have a great weekend!
 Aizhamal

 On Tue, May 21, 2019 at 8:37 PM Kenneth Knowles 
 wrote:

> Great idea.
>
> Austin - point well taken about whether the PMC really has to
> micro-manage here. The stakes are potentially very high, but so are the
> stakes for code and website changes.
>
> I know that comdev votes authoring privileges to people who are not
> committers, but they are not speaking on behalf of comdev but under their
> own name.
>
> Let's definitely find a way to be effective on social media.
>
> Kenn
>
> On Tue, May 21, 2019 at 4:14 AM Maximilian Michels 
> wrote:
>
>> Hi Aizhamal,
>>
>> This is a great idea. I think it would help Beam to be more prominent
>> on
>> social media.
>>
>> We need to discuss this also on the private@ mailing list but I
>> don't
>> see anything standing in the way if the PMC always gets to approve
>> the
>> proposed social media postings.
>>
>> I could even imagine that the PMC gives rights to a Beam community
>> member to post in their name.
>>
>> Thanks,
>> Max
>>
>> On 21.05.19 03:09, Austin Bennett wrote:
>> > Is PMC definitely in charge of this (approving, communication
>> channel,
>> > etc)?
>> >
>> > There could even be a more concrete pull-request-like function even
>> for
>> > things like tweets (to minimize cut/paste operations)?
>> >
>> > I remember a bit of a mechanism having been proposed some time ago
>> (in
>> > another circumstance), though doesn't look like it made it terribly
>> far:
>> >
>> http://www.redhenlab.org/home/the-cognitive-core-research-topics-in-red-hen/the-barnyard/-slick-tweeting
>> > (I haven't otherwise seen such functionality).
>> >
>> >
>> >
>> > On Mon, May 20, 2019 at 4:54 PM Robert Burke > > > wrote:
>> >
>> > +1
>> > As a twitter user, I like this idea.
>> >
>> > On Mon, 20 May 2019 at 15:18, Aizhamal Nurmamat kyzy
>> > mailto:aizha...@google.com>> wrote:
>> >
>> > Hello everyone,
>> >
>> >
>> > What does the community think of making Apache Beam’s social
>> > media presence more active and more community driven?
>> >
>> >
>> > The Slack and StackOverflow for Apache Beam offer pretty
>> nice
>> > support, but we still could utilize Twitter & LinkedIn
>> better to
>> > share more interesting Beam news. For example, we could
>> tweet to
>> > welcome new committers, announce new features consistently,
>> > share and recognize contributions, promote events and
>> meetups,
>> > share other news that are relevant to Beam, big data, etc.
>> >
>> >
>> > I understand that PMC members may not have time to do
>> curation,
>> > moderation and creation of content; so I was wondering if we
>> > could create a spreadsheet where community members could
>> propose

Re: Plan for dropping python 2 support

[Robert Bradshaw]

Until Python 3 support for Beam is officially out of beta and
recommended, I don't think we can tell people to stop using Python 2.
Given that 2020 is just over 6 months away, that seems a short
transition time, so I would guess we'll have to continue supporting
Python 2 sometime into 2020.

A quick survey of users would be valuable here. But first priority is
making Python 3 rock solid so we can unconditionally recommend it over
Python 2.

On Wed, Jun 5, 2019 at 12:27 PM Ismaël Mejía  wrote:
>
> Python 2 won't be maintained after 2020 [1]. I was wondering what will
> be our (Beam) plan for this. Other projects [2] have started to alert
> users that support will be removed so maybe we should decide or policy
> for this too.
>
> [1] https://pythonclock.org/
> [2] https://spark.apache.org/news/plan-for-dropping-python-2-support.html

Plan for dropping python 2 support

[Ismaël Mejía]

Python 2 won't be maintained after 2020 [1]. I was wondering what will
be our (Beam) plan for this. Other projects [2] have started to alert
users that support will be removed so maybe we should decide or policy
for this too.

[1] https://pythonclock.org/
[2] https://spark.apache.org/news/plan-for-dropping-python-2-support.html