Re: [DISCUSS] Using Verbs for Transforms

[Kenneth Knowles]

To be clear: I am not saying that I think the discussion has concluded. I
think we should give some more time for different time zone rotations to
occur. I just meant to say that if it does come to a vote, I'd prefer to
keep it focused rather than generalizing.

On Tue, Oct 25, 2016 at 10:51 PM Kenneth Knowles  wrote:

> I'd prefer to keep the vote focused on this rename, not a general policy.
>
> On Tue, Oct 25, 2016 at 10:26 PM Jean-Baptiste Onofré 
> wrote:
>
> Yes I would start a formal vote with the three proposals: descriptive
> verb, adjective, verbs + adjective.
>
> Regards
> JB
>
> ⁣​
>
> On Oct 26, 2016, 07:16, at 07:16, Jesse Anderson 
> wrote:
> >We need to make a decision on this so Neelesh can finish his commit.
> >Should
> >we take a vote or something?
> >
> >On Tue, Oct 25, 2016, 7:55 AM Jean-Baptiste Onofré 
> >wrote:
> >
> >> Sounds good to me.
> >>
> >> ⁣​
> >>
> >> On Oct 24, 2016, 19:11, at 19:11, je...@smokinghand.com wrote:
> >> >I prefer MakeDistinct if we have to make it a verb.
> >>
>
>

Re: [DISCUSS] Using Verbs for Transforms

[Kenneth Knowles]

I'd prefer to keep the vote focused on this rename, not a general policy.

On Tue, Oct 25, 2016 at 10:26 PM Jean-Baptiste Onofré 
wrote:

> Yes I would start a formal vote with the three proposals: descriptive
> verb, adjective, verbs + adjective.
>
> Regards
> JB
>
> ⁣​
>
> On Oct 26, 2016, 07:16, at 07:16, Jesse Anderson 
> wrote:
> >We need to make a decision on this so Neelesh can finish his commit.
> >Should
> >we take a vote or something?
> >
> >On Tue, Oct 25, 2016, 7:55 AM Jean-Baptiste Onofré 
> >wrote:
> >
> >> Sounds good to me.
> >>
> >> ⁣​
> >>
> >> On Oct 24, 2016, 19:11, at 19:11, je...@smokinghand.com wrote:
> >> >I prefer MakeDistinct if we have to make it a verb.
> >>
>

Re: [DISCUSS] Current ongoing work on runners

[Jean-Baptiste Onofré]

Good idea.

Regards
JB

⁣​

On Oct 25, 2016, 17:57, at 17:57, Aljoscha Krettek  wrote:
>I think we might need to update the capability matrix with some of the
>new
>features that have popped up. Immediate things that come to mind are:
>* Timer/State API for user DoFns (coupled with new-style DoFn) (not yet
>completely in master)
> * SplittableDoFn
>
>This would allow tracking the process in each of these for each runner
>and
>would not require hunting for that information in email threads.
>
>On Tue, 25 Oct 2016 at 08:12 Jean-Baptiste Onofré 
>wrote:
>
>> +1. For me it's one of the most important point for the new website.
>We
>> should give a clear and exhaustive list of what we have, both for
>runners
>> and IOs (with supported features).
>>
>> Regards
>> JB
>>
>> ⁣​
>>
>> On Oct 24, 2016, 21:52, at 21:52, "Ismaël Mejía" 
>> wrote:
>> >Hello,
>> >
>> >I am really happy to see new runners been contributed to our
>community
>> >(e.g. GearPump and Apex recently). We have not discussed a lot about
>> >the
>> >current capabilities of both runners.
>> >
>> >Following the recent discussion about making ongoing work more
>explicit
>> >in
>> >the mailing list, I would like to ask the people involved about the
>> >current
>> >status of them, I think it is important to discuss this (apart of
>> >creating
>> >the given JIRAs + updating the capability matrix docs) because more
>> >people
>> >can eventually jump and give a hand on open issues.
>> >
>> >I remember there was a google doc for the  capabilities of each
>runner,
>> >is
>> >this doc still available (sorry I lost the link). I suppose that
>once
>> >these
>> >ongoing runners mature we can add this doc also to the website.
>> >https://beam.apache.org/learn/runners/capability-matrix/
>> >
>> >Regards,
>> >Ismaël
>> >
>> >ps. @Amit, given that the spark 2 (Dataset based) runner has also a
>> >feature
>> >branch, if you consider it worth, can you please share a bit about
>that
>> >work too.
>> >
>> >ps2. Can anyone please share the link to the google doc I was
>talking
>> >about, I can't find it after the recent changes to the website.
>> >​
>>

Re: [DISCUSS] Using Verbs for Transforms

[Jean-Baptiste Onofré]

Yes I would start a formal vote with the three proposals: descriptive verb, 
adjective, verbs + adjective.

Regards
JB

⁣​

On Oct 26, 2016, 07:16, at 07:16, Jesse Anderson  wrote:
>We need to make a decision on this so Neelesh can finish his commit.
>Should
>we take a vote or something?
>
>On Tue, Oct 25, 2016, 7:55 AM Jean-Baptiste Onofré 
>wrote:
>
>> Sounds good to me.
>>
>> ⁣​
>>
>> On Oct 24, 2016, 19:11, at 19:11, je...@smokinghand.com wrote:
>> >I prefer MakeDistinct if we have to make it a verb.
>>

Re: [DISCUSS] Using Verbs for Transforms

[Jesse Anderson]

We need to make a decision on this so Neelesh can finish his commit. Should
we take a vote or something?

On Tue, Oct 25, 2016, 7:55 AM Jean-Baptiste Onofré  wrote:

> Sounds good to me.
>
> ⁣​
>
> On Oct 24, 2016, 19:11, at 19:11, je...@smokinghand.com wrote:
> >I prefer MakeDistinct if we have to make it a verb.
>

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Jean-Baptiste Onofré]

Agree. We already discussed about that on the mailing list. I mentionned this 
some weeks ago.

Regards
JB

⁣​

On Oct 26, 2016, 02:26, at 02:26, Dan Halperin  
wrote:
>My reading of the LEGAL threads is that since we are not including
>(shading
>or bundling) the ASL-licensed code we are fine to distribute kinesis-io
>module. This was the original conclusion that LEGAL-198 got to, and
>that
>thread has not been resolved differently (even if Spark went ahead and
>broke the assembly). The beam-sdks-java-io-kinesis module is an
>optional
>part (Beam materially works just fine without it).
>
>So I think we're fine to keep this vote open.
>
>+1 (binding) on the release
>
>Thanks Aljoscha!
>
>
>On Tue, Oct 25, 2016 at 12:07 PM, Aljoscha Krettek
>
>wrote:
>
>> Yep, I was looking at those same threads when I reviewing the
>artefacts.
>> The release was already close to being finished so I went through
>with it
>> but if we think it's not good to have them in we should quickly
>cancel in
>> favour of a new RC without a published Kinesis connector.
>>
>> On Tue, 25 Oct 2016 at 20:46 Dan Halperin
>
>> wrote:
>>
>> > I can't tell whether it is a problem that we are distributing the
>> > beam-sdks-java-io-kinesis module [0].
>> >
>> > Here is the dev@ discussion thread [1] and the (unanswered)
>relevant
>> LEGAL
>> > thread [2].
>> > We linked through to a Spark-related discussion [3], and here is
>how to
>> > disable distribution of the KinesisIO module [4].
>> >
>> > [0]
>> >
>> > https://repository.apache.org/content/repositories/staging/
>> org/apache/beam/beam-sdks-java-io-kinesis/
>> > [1]
>> >
>> > https://lists.apache.org/thread.html/6784bc005f329d93fd59d0f8759ed4
>> 745e72f105e39d869e094d9645@%3Cdev.beam.apache.org%3E
>> > [2]
>> >
>> > https://issues.apache.org/jira/browse/LEGAL-198?
>> focusedCommentId=15471529=com.atlassian.jira.
>> plugin.system.issuetabpanels:comment-tabpanel#comment-15471529
>> > [3] https://issues.apache.org/jira/browse/SPARK-17418
>> > [4] https://github.com/apache/spark/pull/15167/files
>> >
>> > Dan
>> >
>> > On Tue, Oct 25, 2016 at 11:01 AM, Seetharam Venkatesh <
>> > venkat...@innerzeal.com> wrote:
>> >
>> > > +1
>> > >
>> > > Thanks!
>> > >
>> > > On Mon, Oct 24, 2016 at 2:30 PM Aljoscha Krettek
>
>> > > wrote:
>> > >
>> > > > Hi Team!
>> > > >
>> > > > Please review and vote at your leisure on release candidate #1
>for
>> > > version
>> > > > 0.3.0-incubating, as follows:
>> > > > [ ] +1, Approve the release
>> > > > [ ] -1, Do not approve the release (please provide specific
>comments)
>> > > >
>> > > > The complete staging area is available for your review, which
>> includes:
>> > > > * JIRA release notes [1],
>> > > > * the official Apache source release to be deployed to
>> dist.apache.org
>> > > > [2],
>> > > > * all artifacts to be deployed to the Maven Central Repository
>[3],
>> > > > * source code tag "v0.3.0-incubating-RC1" [4],
>> > > > * website pull request listing the release and publishing the
>API
>> > > reference
>> > > > manual [5].
>> > > >
>> > > > Please keep in mind that this release is not focused on
>providing new
>> > > > functionality. We want to refine the release process and make
>stable
>> > > source
>> > > > and binary artefacts available to our users.
>> > > >
>> > > > The vote will be open for at least 72 hours. It is adopted by
>> majority
>> > > > approval, with at least 3 PPMC affirmative votes.
>> > > >
>> > > > Cheers,
>> > > > Aljoscha
>> > > >
>> > > > [1]
>> > > >
>> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> > > projectId=12319527=12338051
>> > > > [2]
>> > > >
>> >
>https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-incubating/
>> > > > [3]
>> > > > https://repository.apache.org/content/repositories/staging/
>> > > org/apache/beam/
>> > > > [4]
>> > > >
>> > > > https://git-wip-us.apache.org/repos/asf?p=incubator-beam.
>> git;a=tag;h=
>> > > 5d86ff7f04862444c266142b0d5acecb5a6b7144
>> > > > [5] https://github.com/apache/incubator-beam-site/pull/52
>> > > >
>> > >
>> >
>>

Re: [DISCUSS] Merging master -> feature branch

[Jean-Baptiste Onofré]

Agree. When possible it would be great to have the branch merged on master 
quickly, even when it's not fully ready. It would give more visibility to 
potential contributors.

Regards
JB

⁣​

On Oct 25, 2016, 23:34, at 23:34, Kenneth Knowles  
wrote:
>Hi all,
>
>While collaborating on the apex-runner branch, the issue of how best to
>continuously merge master into the feature branch came up. IMO it
>differs
>somewhat from normal commits in two notable ways:
>
>1. Modulo fix-ups, it is actually not adding any new code to the
>overall
>codebase, so reviews don't serve to raise the quality bar for
>contributions.
>2. It is pretty important to do very frequently to keep out of trouble,
>so
>friction must be thoroughly justified.
>
>I really wouldn't want reviewing a daily merge from master to consume
>someone's time every day. On the gearpump-runner branch we had some
>major
>review latency problems. Obviously, I'd like to hear from folks on
>other
>feature branches. How has this process been for the Python SDK?
>
>I will also throw out an idea for a variation in process:
>
>1. A committer merges master to their feature branch without
>conflicts.*
>2. They open a PR for the merge.
>3a. If the tests pass without modifications _the committer self-merges
>the
>PR without waiting for review_.
>3b. If there are any adjustments needed, these go in separate commits
>on
>the same PR and the review process is as usual (the reviewer can review
>just the added commits).
>
>What do you think? Does this treat such merges too blithely? This is
>meant
>just as a starting point for discussion.
>
>Kenn
>
>* There should never be real conflicts unless something strange has
>happened - the feature can't be edited on the master branch, and master
>stuff shouldn't be touched on the feature branch.

Re: Apex runner integration tests

[Kenneth Knowles]

I've commented on a PR but also want to respond here.

In the precommit, we run
https://builds.apache.org/job/beam_PreCommit_MavenVerify/ which uses
-Pjenkins-precommit to select very few integration tests. It should just be
unit tests and integration tests based on our examples. This catches the
bulk of issues without being too slow

In postcommit, we also run `mvn verify` in
https://builds.apache.org/job/beam_PostCommit_MavenVerify/ which has a
slightly different configuration. I actually can't decipher the exact level
of overlap.

In postcommit, we also run a separate full RunnableOnService job per
runner:
https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/
,
https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_GearpumpLocal/
,
https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_FlinkLocal/
,
https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_SparkLocal/.
We definitely should add Apex to this list ASAP.

Hope this clears things up a bit.

Kenn

On Tue, Oct 25, 2016 at 10:28 AM Thomas Weise  wrote:

> The Apex runner has the integration tests enabled and that causes Travis PR
> builds to fail with timeout (they complete in Jenkins).
>
> What is the correct setup for this, when are the tests supposed to run?
>
>
> https://github.com/apache/incubator-beam/blob/apex-runner/runners/apex/pom.xml#L190
>
> Thanks,
> Thomas
>

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Kenneth Knowles]

+1 (binding)

On Tue, Oct 25, 2016 at 5:26 PM Dan Halperin 
wrote:

> My reading of the LEGAL threads is that since we are not including (shading
> or bundling) the ASL-licensed code we are fine to distribute kinesis-io
> module. This was the original conclusion that LEGAL-198 got to, and that
> thread has not been resolved differently (even if Spark went ahead and
> broke the assembly). The beam-sdks-java-io-kinesis module is an optional
> part (Beam materially works just fine without it).
>
> So I think we're fine to keep this vote open.
>
> +1 (binding) on the release
>
> Thanks Aljoscha!
>
>
> On Tue, Oct 25, 2016 at 12:07 PM, Aljoscha Krettek 
> wrote:
>
> > Yep, I was looking at those same threads when I reviewing the artefacts.
> > The release was already close to being finished so I went through with it
> > but if we think it's not good to have them in we should quickly cancel in
> > favour of a new RC without a published Kinesis connector.
> >
> > On Tue, 25 Oct 2016 at 20:46 Dan Halperin 
> > wrote:
> >
> > > I can't tell whether it is a problem that we are distributing the
> > > beam-sdks-java-io-kinesis module [0].
> > >
> > > Here is the dev@ discussion thread [1] and the (unanswered) relevant
> > LEGAL
> > > thread [2].
> > > We linked through to a Spark-related discussion [3], and here is how to
> > > disable distribution of the KinesisIO module [4].
> > >
> > > [0]
> > >
> > > https://repository.apache.org/content/repositories/staging/
> > org/apache/beam/beam-sdks-java-io-kinesis/
> > > [1]
> > >
> > > https://lists.apache.org/thread.html/6784bc005f329d93fd59d0f8759ed4
> > 745e72f105e39d869e094d9645@%3Cdev.beam.apache.org%3E
> > > [2]
> > >
> > > https://issues.apache.org/jira/browse/LEGAL-198?
> > focusedCommentId=15471529=com.atlassian.jira.
> > plugin.system.issuetabpanels:comment-tabpanel#comment-15471529
> > > [3] https://issues.apache.org/jira/browse/SPARK-17418
> > > [4] https://github.com/apache/spark/pull/15167/files
> > >
> > > Dan
> > >
> > > On Tue, Oct 25, 2016 at 11:01 AM, Seetharam Venkatesh <
> > > venkat...@innerzeal.com> wrote:
> > >
> > > > +1
> > > >
> > > > Thanks!
> > > >
> > > > On Mon, Oct 24, 2016 at 2:30 PM Aljoscha Krettek <
> aljos...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Team!
> > > > >
> > > > > Please review and vote at your leisure on release candidate #1 for
> > > > version
> > > > > 0.3.0-incubating, as follows:
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > > > >
> > > > > The complete staging area is available for your review, which
> > includes:
> > > > > * JIRA release notes [1],
> > > > > * the official Apache source release to be deployed to
> > dist.apache.org
> > > > > [2],
> > > > > * all artifacts to be deployed to the Maven Central Repository [3],
> > > > > * source code tag "v0.3.0-incubating-RC1" [4],
> > > > > * website pull request listing the release and publishing the API
> > > > reference
> > > > > manual [5].
> > > > >
> > > > > Please keep in mind that this release is not focused on providing
> new
> > > > > functionality. We want to refine the release process and make
> stable
> > > > source
> > > > > and binary artefacts available to our users.
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > approval, with at least 3 PPMC affirmative votes.
> > > > >
> > > > > Cheers,
> > > > > Aljoscha
> > > > >
> > > > > [1]
> > > > >
> > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > projectId=12319527=12338051
> > > > > [2]
> > > > >
> > >
> https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-incubating/
> > > > > [3]
> > > > > https://repository.apache.org/content/repositories/staging/
> > > > org/apache/beam/
> > > > > [4]
> > > > >
> > > > > https://git-wip-us.apache.org/repos/asf?p=incubator-beam.
> > git;a=tag;h=
> > > > 5d86ff7f04862444c266142b0d5acecb5a6b7144
> > > > > [5] https://github.com/apache/incubator-beam-site/pull/52
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Current ongoing work on runners

[Manu Zhang]

We usually have docs sitting together with the source codes such that each
release has its own versioned docs. If the capability matrix is like other
codes, we can update it as we add new features. It also applies to other
docs like new IO. We could make it a requirement in the PR template.

Thanks,
Manu


On Wed, Oct 26, 2016 at 7:24 AM Thomas Weise  wrote:

> I'm planning to take up the discussion about Apex runner current state and
> proposed next steps in a separate thread.
>
> Thanks,
> Thomas
>
>
> On Tue, Oct 25, 2016 at 10:32 AM, Amit Sela  wrote:
>
> > SparkRunner status:
> >
> > V1 (Spark 1.6.x - DStream/RDD API):
> > *Batch*: Full model support for batch, continuous ROS testing setup is in
> > process now so that CI will validate constantly.
> > *Streaming*: Supporting UnboundedSource is in review
> > , starting to work
> on
> > triggers and accumulation modes now.
> >
> > V2 (Spark 2.x - Dataset API):
> > This is on hold for now as Spark 2.0 - Dataset AP for streaming (AKA
> > "Structured Streaming") is marked Alpha.
> > In addition, there are still some basic properties in the Dataset API
> that
> > are missing and will be required to properly support Beam:
> >
> >- Stateful operators.
> >- Encoders (Spark's new schema-based coders) optimization support for
> >classes that are a bit more sophisticated than POJO's (generics, inner
> >classes, etc.).
> >
> > Also waiting to see if Watermarks and purging late/stale data will be
> > introduced in 2.1 (currently the Dataset grows indefinitely which is not
> > something acceptable for production applications).
> > Once this becomes more clear (2.1 release ?) I will get back to working
> on
> > this because in general the Dataset API is preferred as it is actually a
> > real unified API for batch and streaming (and the schema-based
> > optimizations are also interesting).
> >
> > I hope this gives a clear view of the SparkRunner status, feel free to
> ping
> > me for more details on the user/dev list or Slack.
> >
> > Thanks,
> > Amit
> >
> > On Tue, Oct 25, 2016 at 6:57 PM Aljoscha Krettek 
> > wrote:
> >
> > > I think we might need to update the capability matrix with some of the
> > new
> > > features that have popped up. Immediate things that come to mind are:
> > >  * Timer/State API for user DoFns (coupled with new-style DoFn) (not
> yet
> > > completely in master)
> > >  * SplittableDoFn
> > >
> > > This would allow tracking the process in each of these for each runner
> > and
> > > would not require hunting for that information in email threads.
> > >
> > > On Tue, 25 Oct 2016 at 08:12 Jean-Baptiste Onofré 
> > wrote:
> > >
> > > > +1. For me it's one of the most important point for the new website.
> We
> > > > should give a clear and exhaustive list of what we have, both for
> > runners
> > > > and IOs (with supported features).
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > ⁣​
> > > >
> > > > On Oct 24, 2016, 21:52, at 21:52, "Ismaël Mejía" 
> > > > wrote:
> > > > >Hello,
> > > > >
> > > > >I am really happy to see new runners been contributed to our
> community
> > > > >(e.g. GearPump and Apex recently). We have not discussed a lot about
> > > > >the
> > > > >current capabilities of both runners.
> > > > >
> > > > >Following the recent discussion about making ongoing work more
> > explicit
> > > > >in
> > > > >the mailing list, I would like to ask the people involved about the
> > > > >current
> > > > >status of them, I think it is important to discuss this (apart of
> > > > >creating
> > > > >the given JIRAs + updating the capability matrix docs) because more
> > > > >people
> > > > >can eventually jump and give a hand on open issues.
> > > > >
> > > > >I remember there was a google doc for the  capabilities of each
> > runner,
> > > > >is
> > > > >this doc still available (sorry I lost the link). I suppose that
> once
> > > > >these
> > > > >ongoing runners mature we can add this doc also to the website.
> > > > >https://beam.apache.org/learn/runners/capability-matrix/
> > > > >
> > > > >Regards,
> > > > >Ismaël
> > > > >
> > > > >ps. @Amit, given that the spark 2 (Dataset based) runner has also a
> > > > >feature
> > > > >branch, if you consider it worth, can you please share a bit about
> > that
> > > > >work too.
> > > > >
> > > > >ps2. Can anyone please share the link to the google doc I was
> talking
> > > > >about, I can't find it after the recent changes to the website.
> > > > >​
> > > >
> > >
> >
>

Re: [DISCUSS] Merging master -> feature branch

[Manu Zhang]

I like this idea as a maintainer for gearpump-runner branch. Usually the
merge is to keep the feature branch updated to the latest API changes on
master.

Sorry that I haven't made merge PR as frequent as possible which would
bring in a lot of changes each time and make it harder for others to
review. On the other hand, it's not easy for the feature branch to follow
master's steps since master has far more committers/contributors to work
on.

I like this idea but it won't change much for gearpump-runner branch since
we still need committers to merge.

Thanks,
Manu





On Wed, Oct 26, 2016 at 7:58 AM Robert Bradshaw 
wrote:

On Tue, Oct 25, 2016 at 2:33 PM, Kenneth Knowles 
wrote:
> Hi all,
>
> While collaborating on the apex-runner branch, the issue of how best to
> continuously merge master into the feature branch came up. IMO it differs
> somewhat from normal commits in two notable ways:
>
> 1. Modulo fix-ups, it is actually not adding any new code to the overall
> codebase, so reviews don't serve to raise the quality bar for
contributions.
> 2. It is pretty important to do very frequently to keep out of trouble, so
> friction must be thoroughly justified.
>
> I really wouldn't want reviewing a daily merge from master to consume
> someone's time every day. On the gearpump-runner branch we had some major
> review latency problems. Obviously, I'd like to hear from folks on other
> feature branches. How has this process been for the Python SDK?

The Python SDK sits at the extreme end of there being no conflicts,
and due to the paucity of intersection, little motivation to bother
merging at all.

> I will also throw out an idea for a variation in process:
>
> 1. A committer merges master to their feature branch without conflicts.*
> 2. They open a PR for the merge.
> 3a. If the tests pass without modifications _the committer self-merges the
> PR without waiting for review_.
> 3b. If there are any adjustments needed, these go in separate commits on
> the same PR and the review process is as usual (the reviewer can review
> just the added commits).
>
> What do you think? Does this treat such merges too blithely? This is meant
> just as a starting point for discussion.
>
> Kenn
>
> * There should never be real conflicts unless something strange has
> happened - the feature can't be edited on the master branch, and master
> stuff shouldn't be touched on the feature branch.

I think you're being a little optimistic about there rarely being a
need for conflict resolution--often work on a feature requires
refactoring/extending the main codebase. Hopefully as we provide a
clean (and stable) API between runners and SDKs this will still be the
case, but I don't think we're there yet.

If there are conflicts, does one check in the conflict markers then
resolve in a separate commit? Only for messy ones, or all the time?
Certainly if further adjustments are needed we should do a full
review.

In my opinion, reviewing a merge should not be too laborious a process
(e.g. one needn't necessarily read the entire diff as one would with a
standard commit). It shouldn't be blocking anyone to wait for a review
(one can always develop on top of the merge). If there's not enough
activity on a branch to do this, the branch has other troubles. So I'd
lean towards not making an exception here, but could be convinced
otherwise.

- Robert

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Dan Halperin]

My reading of the LEGAL threads is that since we are not including (shading
or bundling) the ASL-licensed code we are fine to distribute kinesis-io
module. This was the original conclusion that LEGAL-198 got to, and that
thread has not been resolved differently (even if Spark went ahead and
broke the assembly). The beam-sdks-java-io-kinesis module is an optional
part (Beam materially works just fine without it).

So I think we're fine to keep this vote open.

+1 (binding) on the release

Thanks Aljoscha!


On Tue, Oct 25, 2016 at 12:07 PM, Aljoscha Krettek 
wrote:

> Yep, I was looking at those same threads when I reviewing the artefacts.
> The release was already close to being finished so I went through with it
> but if we think it's not good to have them in we should quickly cancel in
> favour of a new RC without a published Kinesis connector.
>
> On Tue, 25 Oct 2016 at 20:46 Dan Halperin 
> wrote:
>
> > I can't tell whether it is a problem that we are distributing the
> > beam-sdks-java-io-kinesis module [0].
> >
> > Here is the dev@ discussion thread [1] and the (unanswered) relevant
> LEGAL
> > thread [2].
> > We linked through to a Spark-related discussion [3], and here is how to
> > disable distribution of the KinesisIO module [4].
> >
> > [0]
> >
> > https://repository.apache.org/content/repositories/staging/
> org/apache/beam/beam-sdks-java-io-kinesis/
> > [1]
> >
> > https://lists.apache.org/thread.html/6784bc005f329d93fd59d0f8759ed4
> 745e72f105e39d869e094d9645@%3Cdev.beam.apache.org%3E
> > [2]
> >
> > https://issues.apache.org/jira/browse/LEGAL-198?
> focusedCommentId=15471529=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15471529
> > [3] https://issues.apache.org/jira/browse/SPARK-17418
> > [4] https://github.com/apache/spark/pull/15167/files
> >
> > Dan
> >
> > On Tue, Oct 25, 2016 at 11:01 AM, Seetharam Venkatesh <
> > venkat...@innerzeal.com> wrote:
> >
> > > +1
> > >
> > > Thanks!
> > >
> > > On Mon, Oct 24, 2016 at 2:30 PM Aljoscha Krettek 
> > > wrote:
> > >
> > > > Hi Team!
> > > >
> > > > Please review and vote at your leisure on release candidate #1 for
> > > version
> > > > 0.3.0-incubating, as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release to be deployed to
> dist.apache.org
> > > > [2],
> > > > * all artifacts to be deployed to the Maven Central Repository [3],
> > > > * source code tag "v0.3.0-incubating-RC1" [4],
> > > > * website pull request listing the release and publishing the API
> > > reference
> > > > manual [5].
> > > >
> > > > Please keep in mind that this release is not focused on providing new
> > > > functionality. We want to refine the release process and make stable
> > > source
> > > > and binary artefacts available to our users.
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PPMC affirmative votes.
> > > >
> > > > Cheers,
> > > > Aljoscha
> > > >
> > > > [1]
> > > >
> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > projectId=12319527=12338051
> > > > [2]
> > > >
> > https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-incubating/
> > > > [3]
> > > > https://repository.apache.org/content/repositories/staging/
> > > org/apache/beam/
> > > > [4]
> > > >
> > > > https://git-wip-us.apache.org/repos/asf?p=incubator-beam.
> git;a=tag;h=
> > > 5d86ff7f04862444c266142b0d5acecb5a6b7144
> > > > [5] https://github.com/apache/incubator-beam-site/pull/52
> > > >
> > >
> >
>

Re: The Availability of PipelineOptions

[Thomas Groh]

This proposal is tracked as (only)
https://issues.apache.org/jira/browse/BEAM-818

Prerequisite sub-tasks are tracked as BEAM-826 to BEAM-828. There may be
more uses that I'm missing, but PubSubIO, Write, and BigQueryIO are the
three places I'm sure that options from the Pipeline are used.

On Tue, Oct 25, 2016 at 2:08 PM, Kenneth Knowles 
wrote:

> In the spirit of some recent conversations about tracking proposals like
> this, are there JIRAs you can [file and then] mention on this thread?
>
> On Tue, Oct 25, 2016 at 2:07 PM Kenneth Knowles  wrote:
>
> > Yea +1. Definitely a real prerequisite to a true runner-independent
> graph.
> >
> > On Tue, Oct 25, 2016 at 1:24 PM Amit Sela  wrote:
> >
> > +1
> >
> > On Tue, Oct 25, 2016 at 8:43 PM Robert Bradshaw
> > 
> > wrote:
> >
> > > +1
> > >
> > > On Tue, Oct 25, 2016 at 7:26 AM, Thomas Weise  wrote:
> > > > +1
> > > >
> > > >
> > > > On Tue, Oct 25, 2016 at 3:03 AM, Jean-Baptiste Onofré <
> j...@nanthrax.net
> > >
> > > > wrote:
> > > >
> > > >> +1
> > > >>
> > > >> Agree
> > > >>
> > > >> Regards
> > > >> JB
> > > >>
> > > >> ⁣
> > > >>
> > > >> On Oct 25, 2016, 12:01, at 12:01, Aljoscha Krettek <
> > aljos...@apache.org
> > > >
> > > >> wrote:
> > > >> >+1 This sounds quite straightforward.
> > > >> >
> > > >> >On Tue, 25 Oct 2016 at 01:36 Thomas Groh  >
> > > >> >wrote:
> > > >> >
> > > >> >> Hey everyone,
> > > >> >>
> > > >> >> I've been working on a declaration of intent for how we want to
> use
> > > >> >> PipelineOptions and an API change to be consistent with that
> > intent.
> > > >> >This
> > > >> >> is generally part of the move to the Runner API, specifically the
> > > >> >desire to
> > > >> >> be able to reuse Pipelines and the ability to choose runner at
> the
> > > >> >time of
> > > >> >> the call to run.
> > > >> >>
> > > >> >> The high-level summary is I wan to remove the
> > > >> >Pipeline.getPipelineOptions
> > > >> >> method.
> > > >> >>
> > > >> >> I believe this will be compatible with other in-flight proposals,
> > > >> >> especially Dynamic PipelineOptions, but would love to see what
> > > >> >everyone
> > > >> >> else thinks. The document is available at the link below.
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >https://docs.google.com/document/d/1Wr05cYdqnCfrLLqSk-
> > > >> -XmGMGgDwwNwWZaFbxLKvPqEQ/edit?usp=sharing
> > > >> >>
> > > >> >> Thanks,
> > > >> >>
> > > >> >> Thomas
> > > >> >>
> > > >>
> > >
> >
> >
>

[DISCUSS] Merging master -> feature branch

[Kenneth Knowles]

Hi all,

While collaborating on the apex-runner branch, the issue of how best to
continuously merge master into the feature branch came up. IMO it differs
somewhat from normal commits in two notable ways:

1. Modulo fix-ups, it is actually not adding any new code to the overall
codebase, so reviews don't serve to raise the quality bar for contributions.
2. It is pretty important to do very frequently to keep out of trouble, so
friction must be thoroughly justified.

I really wouldn't want reviewing a daily merge from master to consume
someone's time every day. On the gearpump-runner branch we had some major
review latency problems. Obviously, I'd like to hear from folks on other
feature branches. How has this process been for the Python SDK?

I will also throw out an idea for a variation in process:

1. A committer merges master to their feature branch without conflicts.*
2. They open a PR for the merge.
3a. If the tests pass without modifications _the committer self-merges the
PR without waiting for review_.
3b. If there are any adjustments needed, these go in separate commits on
the same PR and the review process is as usual (the reviewer can review
just the added commits).

What do you think? Does this treat such merges too blithely? This is meant
just as a starting point for discussion.

Kenn

* There should never be real conflicts unless something strange has
happened - the feature can't be edited on the master branch, and master
stuff shouldn't be touched on the feature branch.

Re: The Availability of PipelineOptions

[Kenneth Knowles]

In the spirit of some recent conversations about tracking proposals like
this, are there JIRAs you can [file and then] mention on this thread?

On Tue, Oct 25, 2016 at 2:07 PM Kenneth Knowles  wrote:

> Yea +1. Definitely a real prerequisite to a true runner-independent graph.
>
> On Tue, Oct 25, 2016 at 1:24 PM Amit Sela  wrote:
>
> +1
>
> On Tue, Oct 25, 2016 at 8:43 PM Robert Bradshaw
> 
> wrote:
>
> > +1
> >
> > On Tue, Oct 25, 2016 at 7:26 AM, Thomas Weise  wrote:
> > > +1
> > >
> > >
> > > On Tue, Oct 25, 2016 at 3:03 AM, Jean-Baptiste Onofré  >
> > > wrote:
> > >
> > >> +1
> > >>
> > >> Agree
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> ⁣
> > >>
> > >> On Oct 25, 2016, 12:01, at 12:01, Aljoscha Krettek <
> aljos...@apache.org
> > >
> > >> wrote:
> > >> >+1 This sounds quite straightforward.
> > >> >
> > >> >On Tue, 25 Oct 2016 at 01:36 Thomas Groh 
> > >> >wrote:
> > >> >
> > >> >> Hey everyone,
> > >> >>
> > >> >> I've been working on a declaration of intent for how we want to use
> > >> >> PipelineOptions and an API change to be consistent with that
> intent.
> > >> >This
> > >> >> is generally part of the move to the Runner API, specifically the
> > >> >desire to
> > >> >> be able to reuse Pipelines and the ability to choose runner at the
> > >> >time of
> > >> >> the call to run.
> > >> >>
> > >> >> The high-level summary is I wan to remove the
> > >> >Pipeline.getPipelineOptions
> > >> >> method.
> > >> >>
> > >> >> I believe this will be compatible with other in-flight proposals,
> > >> >> especially Dynamic PipelineOptions, but would love to see what
> > >> >everyone
> > >> >> else thinks. The document is available at the link below.
> > >> >>
> > >> >>
> > >> >>
> > >> >https://docs.google.com/document/d/1Wr05cYdqnCfrLLqSk-
> > >> -XmGMGgDwwNwWZaFbxLKvPqEQ/edit?usp=sharing
> > >> >>
> > >> >> Thanks,
> > >> >>
> > >> >> Thomas
> > >> >>
> > >>
> >
>
>

Re: The Availability of PipelineOptions

[Kenneth Knowles]

Yea +1. Definitely a real prerequisite to a true runner-independent graph.

On Tue, Oct 25, 2016 at 1:24 PM Amit Sela  wrote:

> +1
>
> On Tue, Oct 25, 2016 at 8:43 PM Robert Bradshaw
> 
> wrote:
>
> > +1
> >
> > On Tue, Oct 25, 2016 at 7:26 AM, Thomas Weise  wrote:
> > > +1
> > >
> > >
> > > On Tue, Oct 25, 2016 at 3:03 AM, Jean-Baptiste Onofré  >
> > > wrote:
> > >
> > >> +1
> > >>
> > >> Agree
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> ⁣
> > >>
> > >> On Oct 25, 2016, 12:01, at 12:01, Aljoscha Krettek <
> aljos...@apache.org
> > >
> > >> wrote:
> > >> >+1 This sounds quite straightforward.
> > >> >
> > >> >On Tue, 25 Oct 2016 at 01:36 Thomas Groh 
> > >> >wrote:
> > >> >
> > >> >> Hey everyone,
> > >> >>
> > >> >> I've been working on a declaration of intent for how we want to use
> > >> >> PipelineOptions and an API change to be consistent with that
> intent.
> > >> >This
> > >> >> is generally part of the move to the Runner API, specifically the
> > >> >desire to
> > >> >> be able to reuse Pipelines and the ability to choose runner at the
> > >> >time of
> > >> >> the call to run.
> > >> >>
> > >> >> The high-level summary is I wan to remove the
> > >> >Pipeline.getPipelineOptions
> > >> >> method.
> > >> >>
> > >> >> I believe this will be compatible with other in-flight proposals,
> > >> >> especially Dynamic PipelineOptions, but would love to see what
> > >> >everyone
> > >> >> else thinks. The document is available at the link below.
> > >> >>
> > >> >>
> > >> >>
> > >> >https://docs.google.com/document/d/1Wr05cYdqnCfrLLqSk-
> > >> -XmGMGgDwwNwWZaFbxLKvPqEQ/edit?usp=sharing
> > >> >>
> > >> >> Thanks,
> > >> >>
> > >> >> Thomas
> > >> >>
> > >>
> >
>

Re: The Availability of PipelineOptions

[Amit Sela]

+1

On Tue, Oct 25, 2016 at 8:43 PM Robert Bradshaw 
wrote:

> +1
>
> On Tue, Oct 25, 2016 at 7:26 AM, Thomas Weise  wrote:
> > +1
> >
> >
> > On Tue, Oct 25, 2016 at 3:03 AM, Jean-Baptiste Onofré 
> > wrote:
> >
> >> +1
> >>
> >> Agree
> >>
> >> Regards
> >> JB
> >>
> >> ⁣
> >>
> >> On Oct 25, 2016, 12:01, at 12:01, Aljoscha Krettek  >
> >> wrote:
> >> >+1 This sounds quite straightforward.
> >> >
> >> >On Tue, 25 Oct 2016 at 01:36 Thomas Groh 
> >> >wrote:
> >> >
> >> >> Hey everyone,
> >> >>
> >> >> I've been working on a declaration of intent for how we want to use
> >> >> PipelineOptions and an API change to be consistent with that intent.
> >> >This
> >> >> is generally part of the move to the Runner API, specifically the
> >> >desire to
> >> >> be able to reuse Pipelines and the ability to choose runner at the
> >> >time of
> >> >> the call to run.
> >> >>
> >> >> The high-level summary is I wan to remove the
> >> >Pipeline.getPipelineOptions
> >> >> method.
> >> >>
> >> >> I believe this will be compatible with other in-flight proposals,
> >> >> especially Dynamic PipelineOptions, but would love to see what
> >> >everyone
> >> >> else thinks. The document is available at the link below.
> >> >>
> >> >>
> >> >>
> >> >https://docs.google.com/document/d/1Wr05cYdqnCfrLLqSk-
> >> -XmGMGgDwwNwWZaFbxLKvPqEQ/edit?usp=sharing
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Thomas
> >> >>
> >>
>

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Davor Bonaci]

+1 (binding)

My understanding that Kinesis licensing is not an issue since we don't
redistribute that code ourselves. (I'd also be fine in excluding that code
from the source distribution if deemed necessary.)

On Tue, Oct 25, 2016 at 11:45 AM, Dan Halperin 
wrote:

> I can't tell whether it is a problem that we are distributing the
> beam-sdks-java-io-kinesis module [0].
>
> Here is the dev@ discussion thread [1] and the (unanswered) relevant LEGAL
> thread [2].
> We linked through to a Spark-related discussion [3], and here is how to
> disable distribution of the KinesisIO module [4].
>
> [0]
> https://repository.apache.org/content/repositories/staging/
> org/apache/beam/beam-sdks-java-io-kinesis/
> [1]
> https://lists.apache.org/thread.html/6784bc005f329d93fd59d0f8759ed4
> 745e72f105e39d869e094d9645@%3Cdev.beam.apache.org%3E
> [2]
> https://issues.apache.org/jira/browse/LEGAL-198?focusedCommentId=15471529;
> page=com.atlassian.jira.plugin.system.issuetabpanels:
> comment-tabpanel#comment-15471529
> [3] https://issues.apache.org/jira/browse/SPARK-17418
> [4] https://github.com/apache/spark/pull/15167/files
>
> Dan
>
> On Tue, Oct 25, 2016 at 11:01 AM, Seetharam Venkatesh <
> venkat...@innerzeal.com> wrote:
>
> > +1
> >
> > Thanks!
> >
> > On Mon, Oct 24, 2016 at 2:30 PM Aljoscha Krettek 
> > wrote:
> >
> > > Hi Team!
> > >
> > > Please review and vote at your leisure on release candidate #1 for
> > version
> > > 0.3.0-incubating, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to dist.apache.org
> > > [2],
> > > * all artifacts to be deployed to the Maven Central Repository [3],
> > > * source code tag "v0.3.0-incubating-RC1" [4],
> > > * website pull request listing the release and publishing the API
> > reference
> > > manual [5].
> > >
> > > Please keep in mind that this release is not focused on providing new
> > > functionality. We want to refine the release process and make stable
> > source
> > > and binary artefacts available to our users.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PPMC affirmative votes.
> > >
> > > Cheers,
> > > Aljoscha
> > >
> > > [1]
> > >
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12319527=12338051
> > > [2]
> > > https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-
> incubating/
> > > [3]
> > > https://repository.apache.org/content/repositories/staging/
> > org/apache/beam/
> > > [4]
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=
> > 5d86ff7f04862444c266142b0d5acecb5a6b7144
> > > [5] https://github.com/apache/incubator-beam-site/pull/52
> > >
> >
>

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Dan Halperin]

I can't tell whether it is a problem that we are distributing the
beam-sdks-java-io-kinesis module [0].

Here is the dev@ discussion thread [1] and the (unanswered) relevant LEGAL
thread [2].
We linked through to a Spark-related discussion [3], and here is how to
disable distribution of the KinesisIO module [4].

[0]
https://repository.apache.org/content/repositories/staging/org/apache/beam/beam-sdks-java-io-kinesis/
[1]
https://lists.apache.org/thread.html/6784bc005f329d93fd59d0f8759ed4745e72f105e39d869e094d9645@%3Cdev.beam.apache.org%3E
[2]
https://issues.apache.org/jira/browse/LEGAL-198?focusedCommentId=15471529=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15471529
[3] https://issues.apache.org/jira/browse/SPARK-17418
[4] https://github.com/apache/spark/pull/15167/files

Dan

On Tue, Oct 25, 2016 at 11:01 AM, Seetharam Venkatesh <
venkat...@innerzeal.com> wrote:

> +1
>
> Thanks!
>
> On Mon, Oct 24, 2016 at 2:30 PM Aljoscha Krettek 
> wrote:
>
> > Hi Team!
> >
> > Please review and vote at your leisure on release candidate #1 for
> version
> > 0.3.0-incubating, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > [2],
> > * all artifacts to be deployed to the Maven Central Repository [3],
> > * source code tag "v0.3.0-incubating-RC1" [4],
> > * website pull request listing the release and publishing the API
> reference
> > manual [5].
> >
> > Please keep in mind that this release is not focused on providing new
> > functionality. We want to refine the release process and make stable
> source
> > and binary artefacts available to our users.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PPMC affirmative votes.
> >
> > Cheers,
> > Aljoscha
> >
> > [1]
> >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527=12338051
> > [2]
> > https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-incubating/
> > [3]
> > https://repository.apache.org/content/repositories/staging/
> org/apache/beam/
> > [4]
> >
> > https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=
> 5d86ff7f04862444c266142b0d5acecb5a6b7144
> > [5] https://github.com/apache/incubator-beam-site/pull/52
> >
>

Re: [DISCUSS] Current ongoing work on runners

[Amit Sela]

SparkRunner status:

V1 (Spark 1.6.x - DStream/RDD API):
*Batch*: Full model support for batch, continuous ROS testing setup is in
process now so that CI will validate constantly.
*Streaming*: Supporting UnboundedSource is in review
, starting to work on
triggers and accumulation modes now.

V2 (Spark 2.x - Dataset API):
This is on hold for now as Spark 2.0 - Dataset AP for streaming (AKA
"Structured Streaming") is marked Alpha.
In addition, there are still some basic properties in the Dataset API that
are missing and will be required to properly support Beam:

   - Stateful operators.
   - Encoders (Spark's new schema-based coders) optimization support for
   classes that are a bit more sophisticated than POJO's (generics, inner
   classes, etc.).

Also waiting to see if Watermarks and purging late/stale data will be
introduced in 2.1 (currently the Dataset grows indefinitely which is not
something acceptable for production applications).
Once this becomes more clear (2.1 release ?) I will get back to working on
this because in general the Dataset API is preferred as it is actually a
real unified API for batch and streaming (and the schema-based
optimizations are also interesting).

I hope this gives a clear view of the SparkRunner status, feel free to ping
me for more details on the user/dev list or Slack.

Thanks,
Amit

On Tue, Oct 25, 2016 at 6:57 PM Aljoscha Krettek 
wrote:

> I think we might need to update the capability matrix with some of the new
> features that have popped up. Immediate things that come to mind are:
>  * Timer/State API for user DoFns (coupled with new-style DoFn) (not yet
> completely in master)
>  * SplittableDoFn
>
> This would allow tracking the process in each of these for each runner and
> would not require hunting for that information in email threads.
>
> On Tue, 25 Oct 2016 at 08:12 Jean-Baptiste Onofré  wrote:
>
> > +1. For me it's one of the most important point for the new website. We
> > should give a clear and exhaustive list of what we have, both for runners
> > and IOs (with supported features).
> >
> > Regards
> > JB
> >
> > ⁣​
> >
> > On Oct 24, 2016, 21:52, at 21:52, "Ismaël Mejía" 
> > wrote:
> > >Hello,
> > >
> > >I am really happy to see new runners been contributed to our community
> > >(e.g. GearPump and Apex recently). We have not discussed a lot about
> > >the
> > >current capabilities of both runners.
> > >
> > >Following the recent discussion about making ongoing work more explicit
> > >in
> > >the mailing list, I would like to ask the people involved about the
> > >current
> > >status of them, I think it is important to discuss this (apart of
> > >creating
> > >the given JIRAs + updating the capability matrix docs) because more
> > >people
> > >can eventually jump and give a hand on open issues.
> > >
> > >I remember there was a google doc for the  capabilities of each runner,
> > >is
> > >this doc still available (sorry I lost the link). I suppose that once
> > >these
> > >ongoing runners mature we can add this doc also to the website.
> > >https://beam.apache.org/learn/runners/capability-matrix/
> > >
> > >Regards,
> > >Ismaël
> > >
> > >ps. @Amit, given that the spark 2 (Dataset based) runner has also a
> > >feature
> > >branch, if you consider it worth, can you please share a bit about that
> > >work too.
> > >
> > >ps2. Can anyone please share the link to the google doc I was talking
> > >about, I can't find it after the recent changes to the website.
> > >​
> >
>

Apex runner integration tests

[Thomas Weise]

The Apex runner has the integration tests enabled and that causes Travis PR
builds to fail with timeout (they complete in Jenkins).

What is the correct setup for this, when are the tests supposed to run?

https://github.com/apache/incubator-beam/blob/apex-runner/runners/apex/pom.xml#L190

Thanks,
Thomas

Re: [DISCUSS] Current ongoing work on runners

[Aljoscha Krettek]

I think we might need to update the capability matrix with some of the new
features that have popped up. Immediate things that come to mind are:
 * Timer/State API for user DoFns (coupled with new-style DoFn) (not yet
completely in master)
 * SplittableDoFn

This would allow tracking the process in each of these for each runner and
would not require hunting for that information in email threads.

On Tue, 25 Oct 2016 at 08:12 Jean-Baptiste Onofré  wrote:

> +1. For me it's one of the most important point for the new website. We
> should give a clear and exhaustive list of what we have, both for runners
> and IOs (with supported features).
>
> Regards
> JB
>
> ⁣​
>
> On Oct 24, 2016, 21:52, at 21:52, "Ismaël Mejía" 
> wrote:
> >Hello,
> >
> >I am really happy to see new runners been contributed to our community
> >(e.g. GearPump and Apex recently). We have not discussed a lot about
> >the
> >current capabilities of both runners.
> >
> >Following the recent discussion about making ongoing work more explicit
> >in
> >the mailing list, I would like to ask the people involved about the
> >current
> >status of them, I think it is important to discuss this (apart of
> >creating
> >the given JIRAs + updating the capability matrix docs) because more
> >people
> >can eventually jump and give a hand on open issues.
> >
> >I remember there was a google doc for the  capabilities of each runner,
> >is
> >this doc still available (sorry I lost the link). I suppose that once
> >these
> >ongoing runners mature we can add this doc also to the website.
> >https://beam.apache.org/learn/runners/capability-matrix/
> >
> >Regards,
> >Ismaël
> >
> >ps. @Amit, given that the spark 2 (Dataset based) runner has also a
> >feature
> >branch, if you consider it worth, can you please share a bit about that
> >work too.
> >
> >ps2. Can anyone please share the link to the google doc I was talking
> >about, I can't find it after the recent changes to the website.
> >​
>

Re: The Availability of PipelineOptions

[Thomas Weise]

+1


On Tue, Oct 25, 2016 at 3:03 AM, Jean-Baptiste Onofré 
wrote:

> +1
>
> Agree
>
> Regards
> JB
>
> ⁣​
>
> On Oct 25, 2016, 12:01, at 12:01, Aljoscha Krettek 
> wrote:
> >+1 This sounds quite straightforward.
> >
> >On Tue, 25 Oct 2016 at 01:36 Thomas Groh 
> >wrote:
> >
> >> Hey everyone,
> >>
> >> I've been working on a declaration of intent for how we want to use
> >> PipelineOptions and an API change to be consistent with that intent.
> >This
> >> is generally part of the move to the Runner API, specifically the
> >desire to
> >> be able to reuse Pipelines and the ability to choose runner at the
> >time of
> >> the call to run.
> >>
> >> The high-level summary is I wan to remove the
> >Pipeline.getPipelineOptions
> >> method.
> >>
> >> I believe this will be compatible with other in-flight proposals,
> >> especially Dynamic PipelineOptions, but would love to see what
> >everyone
> >> else thinks. The document is available at the link below.
> >>
> >>
> >>
> >https://docs.google.com/document/d/1Wr05cYdqnCfrLLqSk-
> -XmGMGgDwwNwWZaFbxLKvPqEQ/edit?usp=sharing
> >>
> >> Thanks,
> >>
> >> Thomas
> >>
>

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Jean-Baptiste Onofré]

Thanks Sergio ;)

Just tried to explain to the others what is a binding vote ;)

Regards
JB

⁣​

On Oct 25, 2016, 11:53, at 11:53, "Sergio Fernández"  wrote:
>On Tue, Oct 25, 2016 at 11:36 AM, Jean-Baptiste Onofré
>
>wrote:
>
>> By the way, your vote is not binding from a podling perspective (you
>are
>> not PPMC). Your vote is binding from IPMC perspective (so when you
>will
>> vote on the incubator mailing list).
>>
>
>Well, PPMC are never binding votes, only IPMC are actually binding.
>That
>I'm not part of the PPMC is not much relevant. Therefore I think my
>vote is
>still a valid binding one; but I can vote again on general@incubator,
>no
>problem.
>
>Sorry for jumping-in too early. Besides a IPMC, I'm also a developer
>interested in Beam ;-)
>
>Cheers,
>
>
>
>
>> On Oct 25, 2016, 11:33, at 11:33, "Sergio Fernández"
>
>> wrote:
>> >+1 (binding)
>> >
>> >So far I've successfully checked:
>> >* signatures and digests
>> >* source releases file layouts
>> >* matched git tags and commit ids
>> >* incubator suffix and disclaimer
>> >* NOTICE and LICENSE files
>> >* license headers
>> >* clean build (Java 1.8.0_91, Scala, 2.11.7, SBT 0.13.9, Debian
>amd64)
>> >
>> >
>> >Couple of minor issues I've seen it'd be great to have fixed in
>> >upcoming
>> >releases:
>> >* MongoDbIOTest fails (addr already in use) when a Mongo service is
>> >locally
>> >running. I'd say the port should be random in the test suite.
>> >* How did you generated the checksums? Because both SHA1/MD5 can't
>be
>> >automatically checked because "no properly formatted SHA1/MD5
>checksum
>> >lines found".
>> >
>> >Great to see the project moving forward at this speed :-)
>> >
>> >Cheers,
>> >
>> >
>> >
>> >On Mon, Oct 24, 2016 at 11:30 PM, Aljoscha Krettek
>> >
>> >wrote:
>> >
>> >> Hi Team!
>> >>
>> >> Please review and vote at your leisure on release candidate #1 for
>> >version
>> >> 0.3.0-incubating, as follows:
>> >> [ ] +1, Approve the release
>> >> [ ] -1, Do not approve the release (please provide specific
>comments)
>> >>
>> >> The complete staging area is available for your review, which
>> >includes:
>> >> * JIRA release notes [1],
>> >> * the official Apache source release to be deployed to
>> >dist.apache.org
>> >> [2],
>> >> * all artifacts to be deployed to the Maven Central Repository
>[3],
>> >> * source code tag "v0.3.0-incubating-RC1" [4],
>> >> * website pull request listing the release and publishing the API
>> >reference
>> >> manual [5].
>> >>
>> >> Please keep in mind that this release is not focused on providing
>new
>> >> functionality. We want to refine the release process and make
>stable
>> >source
>> >> and binary artefacts available to our users.
>> >>
>> >> The vote will be open for at least 72 hours. It is adopted by
>> >majority
>> >> approval, with at least 3 PPMC affirmative votes.
>> >>
>> >> Cheers,
>> >> Aljoscha
>> >>
>> >> [1]
>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> >> projectId=12319527=12338051
>> >> [2] https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-
>> >> incubating/
>> >> [3]
>> >> https://repository.apache.org/content/repositories/staging/
>> >> org/apache/beam/
>> >> [4]
>> >>
>https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=
>> >> 5d86ff7f04862444c266142b0d5acecb5a6b7144
>> >> [5] https://github.com/apache/incubator-beam-site/pull/52
>> >>
>> >
>> >
>> >
>> >--
>> >Sergio Fernández
>> >Partner Technology Manager
>> >Redlink GmbH
>> >m: +43 6602747925
>> >e: sergio.fernan...@redlink.co
>> >w: http://redlink.co
>>
>
>
>
>-- 
>Sergio Fernández
>Partner Technology Manager
>Redlink GmbH
>m: +43 6602747925
>e: sergio.fernan...@redlink.co
>w: http://redlink.co

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Jean-Baptiste Onofré]

Imho it's better to use md5sun/sha1sum.

About mongodb I will implement a free port use to avoid the failing test if an 
existing instance is already running locally.

Regards
JB

⁣​

On Oct 25, 2016, 12:07, at 12:07, Aljoscha Krettek  wrote:
>Hi Sergio,
>I used this line from the Apache release signing doc (
>https://www.apache.org/dev/release-signing.html#md5):
>$ gpg --print-md MD5 [fileName] > [fileName].md5
>
>What is normally used to checksum/verify the md5 and sha hash? As it is
>now, they can be manually verified by comparing the checksum in the
>file
>with one that was derived from the zip. I hope that's still ok for the
>release to go through?
>
>Cheers,
>Aljoscha
>
>On Tue, 25 Oct 2016 at 11:53 Sergio Fernández 
>wrote:
>
>> On Tue, Oct 25, 2016 at 11:36 AM, Jean-Baptiste Onofré
>
>> wrote:
>>
>> > By the way, your vote is not binding from a podling perspective
>(you are
>> > not PPMC). Your vote is binding from IPMC perspective (so when you
>will
>> > vote on the incubator mailing list).
>> >
>>
>> Well, PPMC are never binding votes, only IPMC are actually binding.
>That
>> I'm not part of the PPMC is not much relevant. Therefore I think my
>vote is
>> still a valid binding one; but I can vote again on general@incubator,
>no
>> problem.
>>
>> Sorry for jumping-in too early. Besides a IPMC, I'm also a developer
>> interested in Beam ;-)
>>
>> Cheers,
>>
>>
>>
>>
>> > On Oct 25, 2016, 11:33, at 11:33, "Sergio Fernández"
>
>> > wrote:
>> > >+1 (binding)
>> > >
>> > >So far I've successfully checked:
>> > >* signatures and digests
>> > >* source releases file layouts
>> > >* matched git tags and commit ids
>> > >* incubator suffix and disclaimer
>> > >* NOTICE and LICENSE files
>> > >* license headers
>> > >* clean build (Java 1.8.0_91, Scala, 2.11.7, SBT 0.13.9, Debian
>amd64)
>> > >
>> > >
>> > >Couple of minor issues I've seen it'd be great to have fixed in
>> > >upcoming
>> > >releases:
>> > >* MongoDbIOTest fails (addr already in use) when a Mongo service
>is
>> > >locally
>> > >running. I'd say the port should be random in the test suite.
>> > >* How did you generated the checksums? Because both SHA1/MD5 can't
>be
>> > >automatically checked because "no properly formatted SHA1/MD5
>checksum
>> > >lines found".
>> > >
>> > >Great to see the project moving forward at this speed :-)
>> > >
>> > >Cheers,
>> > >
>> > >
>> > >
>> > >On Mon, Oct 24, 2016 at 11:30 PM, Aljoscha Krettek
>> > >
>> > >wrote:
>> > >
>> > >> Hi Team!
>> > >>
>> > >> Please review and vote at your leisure on release candidate #1
>for
>> > >version
>> > >> 0.3.0-incubating, as follows:
>> > >> [ ] +1, Approve the release
>> > >> [ ] -1, Do not approve the release (please provide specific
>comments)
>> > >>
>> > >> The complete staging area is available for your review, which
>> > >includes:
>> > >> * JIRA release notes [1],
>> > >> * the official Apache source release to be deployed to
>> > >dist.apache.org
>> > >> [2],
>> > >> * all artifacts to be deployed to the Maven Central Repository
>[3],
>> > >> * source code tag "v0.3.0-incubating-RC1" [4],
>> > >> * website pull request listing the release and publishing the
>API
>> > >reference
>> > >> manual [5].
>> > >>
>> > >> Please keep in mind that this release is not focused on
>providing new
>> > >> functionality. We want to refine the release process and make
>stable
>> > >source
>> > >> and binary artefacts available to our users.
>> > >>
>> > >> The vote will be open for at least 72 hours. It is adopted by
>> > >majority
>> > >> approval, with at least 3 PPMC affirmative votes.
>> > >>
>> > >> Cheers,
>> > >> Aljoscha
>> > >>
>> > >> [1]
>> > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> > >> projectId=12319527=12338051
>> > >> [2] https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-
>> > >> incubating/
>> > >> [3]
>> > >> https://repository.apache.org/content/repositories/staging/
>> > >> org/apache/beam/
>> > >> [4]
>> > >>
>https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=
>> > >> 5d86ff7f04862444c266142b0d5acecb5a6b7144
>> > >> [5] https://github.com/apache/incubator-beam-site/pull/52
>> > >>
>> > >
>> > >
>> > >
>> > >--
>> > >Sergio Fernández
>> > >Partner Technology Manager
>> > >Redlink GmbH
>> > >m: +43 6602747925 <+43%20660%202747925>
>> > >e: sergio.fernan...@redlink.co
>> > >w: http://redlink.co
>> >
>>
>>
>>
>> --
>> Sergio Fernández
>> Partner Technology Manager
>> Redlink GmbH
>> m: +43 6602747925 <+43%20660%202747925>
>> e: sergio.fernan...@redlink.co
>> w: http://redlink.co
>>

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Aljoscha Krettek]

Hi Sergio,
I used this line from the Apache release signing doc (
https://www.apache.org/dev/release-signing.html#md5):
$ gpg --print-md MD5 [fileName] > [fileName].md5

What is normally used to checksum/verify the md5 and sha hash? As it is
now, they can be manually verified by comparing the checksum in the file
with one that was derived from the zip. I hope that's still ok for the
release to go through?

Cheers,
Aljoscha

On Tue, 25 Oct 2016 at 11:53 Sergio Fernández  wrote:

> On Tue, Oct 25, 2016 at 11:36 AM, Jean-Baptiste Onofré 
> wrote:
>
> > By the way, your vote is not binding from a podling perspective (you are
> > not PPMC). Your vote is binding from IPMC perspective (so when you will
> > vote on the incubator mailing list).
> >
>
> Well, PPMC are never binding votes, only IPMC are actually binding. That
> I'm not part of the PPMC is not much relevant. Therefore I think my vote is
> still a valid binding one; but I can vote again on general@incubator, no
> problem.
>
> Sorry for jumping-in too early. Besides a IPMC, I'm also a developer
> interested in Beam ;-)
>
> Cheers,
>
>
>
>
> > On Oct 25, 2016, 11:33, at 11:33, "Sergio Fernández" 
> > wrote:
> > >+1 (binding)
> > >
> > >So far I've successfully checked:
> > >* signatures and digests
> > >* source releases file layouts
> > >* matched git tags and commit ids
> > >* incubator suffix and disclaimer
> > >* NOTICE and LICENSE files
> > >* license headers
> > >* clean build (Java 1.8.0_91, Scala, 2.11.7, SBT 0.13.9, Debian amd64)
> > >
> > >
> > >Couple of minor issues I've seen it'd be great to have fixed in
> > >upcoming
> > >releases:
> > >* MongoDbIOTest fails (addr already in use) when a Mongo service is
> > >locally
> > >running. I'd say the port should be random in the test suite.
> > >* How did you generated the checksums? Because both SHA1/MD5 can't be
> > >automatically checked because "no properly formatted SHA1/MD5 checksum
> > >lines found".
> > >
> > >Great to see the project moving forward at this speed :-)
> > >
> > >Cheers,
> > >
> > >
> > >
> > >On Mon, Oct 24, 2016 at 11:30 PM, Aljoscha Krettek
> > >
> > >wrote:
> > >
> > >> Hi Team!
> > >>
> > >> Please review and vote at your leisure on release candidate #1 for
> > >version
> > >> 0.3.0-incubating, as follows:
> > >> [ ] +1, Approve the release
> > >> [ ] -1, Do not approve the release (please provide specific comments)
> > >>
> > >> The complete staging area is available for your review, which
> > >includes:
> > >> * JIRA release notes [1],
> > >> * the official Apache source release to be deployed to
> > >dist.apache.org
> > >> [2],
> > >> * all artifacts to be deployed to the Maven Central Repository [3],
> > >> * source code tag "v0.3.0-incubating-RC1" [4],
> > >> * website pull request listing the release and publishing the API
> > >reference
> > >> manual [5].
> > >>
> > >> Please keep in mind that this release is not focused on providing new
> > >> functionality. We want to refine the release process and make stable
> > >source
> > >> and binary artefacts available to our users.
> > >>
> > >> The vote will be open for at least 72 hours. It is adopted by
> > >majority
> > >> approval, with at least 3 PPMC affirmative votes.
> > >>
> > >> Cheers,
> > >> Aljoscha
> > >>
> > >> [1]
> > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > >> projectId=12319527=12338051
> > >> [2] https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-
> > >> incubating/
> > >> [3]
> > >> https://repository.apache.org/content/repositories/staging/
> > >> org/apache/beam/
> > >> [4]
> > >> https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=
> > >> 5d86ff7f04862444c266142b0d5acecb5a6b7144
> > >> [5] https://github.com/apache/incubator-beam-site/pull/52
> > >>
> > >
> > >
> > >
> > >--
> > >Sergio Fernández
> > >Partner Technology Manager
> > >Redlink GmbH
> > >m: +43 6602747925 <+43%20660%202747925>
> > >e: sergio.fernan...@redlink.co
> > >w: http://redlink.co
> >
>
>
>
> --
> Sergio Fernández
> Partner Technology Manager
> Redlink GmbH
> m: +43 6602747925 <+43%20660%202747925>
> e: sergio.fernan...@redlink.co
> w: http://redlink.co
>

Re: The Availability of PipelineOptions

[Aljoscha Krettek]

+1 This sounds quite straightforward.

On Tue, 25 Oct 2016 at 01:36 Thomas Groh  wrote:

> Hey everyone,
>
> I've been working on a declaration of intent for how we want to use
> PipelineOptions and an API change to be consistent with that intent. This
> is generally part of the move to the Runner API, specifically the desire to
> be able to reuse Pipelines and the ability to choose runner at the time of
> the call to run.
>
> The high-level summary is I wan to remove the Pipeline.getPipelineOptions
> method.
>
> I believe this will be compatible with other in-flight proposals,
> especially Dynamic PipelineOptions, but would love to see what everyone
> else thinks. The document is available at the link below.
>
>
> https://docs.google.com/document/d/1Wr05cYdqnCfrLLqSk--XmGMGgDwwNwWZaFbxLKvPqEQ/edit?usp=sharing
>
> Thanks,
>
> Thomas
>

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Sergio Fernández]

On Tue, Oct 25, 2016 at 11:36 AM, Jean-Baptiste Onofré 
wrote:

> By the way, your vote is not binding from a podling perspective (you are
> not PPMC). Your vote is binding from IPMC perspective (so when you will
> vote on the incubator mailing list).
>

Well, PPMC are never binding votes, only IPMC are actually binding. That
I'm not part of the PPMC is not much relevant. Therefore I think my vote is
still a valid binding one; but I can vote again on general@incubator, no
problem.

Sorry for jumping-in too early. Besides a IPMC, I'm also a developer
interested in Beam ;-)

Cheers,




> On Oct 25, 2016, 11:33, at 11:33, "Sergio Fernández" 
> wrote:
> >+1 (binding)
> >
> >So far I've successfully checked:
> >* signatures and digests
> >* source releases file layouts
> >* matched git tags and commit ids
> >* incubator suffix and disclaimer
> >* NOTICE and LICENSE files
> >* license headers
> >* clean build (Java 1.8.0_91, Scala, 2.11.7, SBT 0.13.9, Debian amd64)
> >
> >
> >Couple of minor issues I've seen it'd be great to have fixed in
> >upcoming
> >releases:
> >* MongoDbIOTest fails (addr already in use) when a Mongo service is
> >locally
> >running. I'd say the port should be random in the test suite.
> >* How did you generated the checksums? Because both SHA1/MD5 can't be
> >automatically checked because "no properly formatted SHA1/MD5 checksum
> >lines found".
> >
> >Great to see the project moving forward at this speed :-)
> >
> >Cheers,
> >
> >
> >
> >On Mon, Oct 24, 2016 at 11:30 PM, Aljoscha Krettek
> >
> >wrote:
> >
> >> Hi Team!
> >>
> >> Please review and vote at your leisure on release candidate #1 for
> >version
> >> 0.3.0-incubating, as follows:
> >> [ ] +1, Approve the release
> >> [ ] -1, Do not approve the release (please provide specific comments)
> >>
> >> The complete staging area is available for your review, which
> >includes:
> >> * JIRA release notes [1],
> >> * the official Apache source release to be deployed to
> >dist.apache.org
> >> [2],
> >> * all artifacts to be deployed to the Maven Central Repository [3],
> >> * source code tag "v0.3.0-incubating-RC1" [4],
> >> * website pull request listing the release and publishing the API
> >reference
> >> manual [5].
> >>
> >> Please keep in mind that this release is not focused on providing new
> >> functionality. We want to refine the release process and make stable
> >source
> >> and binary artefacts available to our users.
> >>
> >> The vote will be open for at least 72 hours. It is adopted by
> >majority
> >> approval, with at least 3 PPMC affirmative votes.
> >>
> >> Cheers,
> >> Aljoscha
> >>
> >> [1]
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >> projectId=12319527=12338051
> >> [2] https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-
> >> incubating/
> >> [3]
> >> https://repository.apache.org/content/repositories/staging/
> >> org/apache/beam/
> >> [4]
> >> https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=
> >> 5d86ff7f04862444c266142b0d5acecb5a6b7144
> >> [5] https://github.com/apache/incubator-beam-site/pull/52
> >>
> >
> >
> >
> >--
> >Sergio Fernández
> >Partner Technology Manager
> >Redlink GmbH
> >m: +43 6602747925
> >e: sergio.fernan...@redlink.co
> >w: http://redlink.co
>



-- 
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernan...@redlink.co
w: http://redlink.co

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Jean-Baptiste Onofré]

By the way, your vote is not binding from a podling perspective (you are not 
PPMC). Your vote is binding from IPMC perspective (so when you will vote on the 
incubator mailing list).

So this vote is +1 (non binding).

Regards
JB

⁣​

On Oct 25, 2016, 11:33, at 11:33, "Sergio Fernández"  wrote:
>+1 (binding)
>
>So far I've successfully checked:
>* signatures and digests
>* source releases file layouts
>* matched git tags and commit ids
>* incubator suffix and disclaimer
>* NOTICE and LICENSE files
>* license headers
>* clean build (Java 1.8.0_91, Scala, 2.11.7, SBT 0.13.9, Debian amd64)
>
>
>Couple of minor issues I've seen it'd be great to have fixed in
>upcoming
>releases:
>* MongoDbIOTest fails (addr already in use) when a Mongo service is
>locally
>running. I'd say the port should be random in the test suite.
>* How did you generated the checksums? Because both SHA1/MD5 can't be
>automatically checked because "no properly formatted SHA1/MD5 checksum
>lines found".
>
>Great to see the project moving forward at this speed :-)
>
>Cheers,
>
>
>
>On Mon, Oct 24, 2016 at 11:30 PM, Aljoscha Krettek
>
>wrote:
>
>> Hi Team!
>>
>> Please review and vote at your leisure on release candidate #1 for
>version
>> 0.3.0-incubating, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> The complete staging area is available for your review, which
>includes:
>> * JIRA release notes [1],
>> * the official Apache source release to be deployed to
>dist.apache.org
>> [2],
>> * all artifacts to be deployed to the Maven Central Repository [3],
>> * source code tag "v0.3.0-incubating-RC1" [4],
>> * website pull request listing the release and publishing the API
>reference
>> manual [5].
>>
>> Please keep in mind that this release is not focused on providing new
>> functionality. We want to refine the release process and make stable
>source
>> and binary artefacts available to our users.
>>
>> The vote will be open for at least 72 hours. It is adopted by
>majority
>> approval, with at least 3 PPMC affirmative votes.
>>
>> Cheers,
>> Aljoscha
>>
>> [1]
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12319527=12338051
>> [2] https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-
>> incubating/
>> [3]
>> https://repository.apache.org/content/repositories/staging/
>> org/apache/beam/
>> [4]
>> https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=
>> 5d86ff7f04862444c266142b0d5acecb5a6b7144
>> [5] https://github.com/apache/incubator-beam-site/pull/52
>>
>
>
>
>-- 
>Sergio Fernández
>Partner Technology Manager
>Redlink GmbH
>m: +43 6602747925
>e: sergio.fernan...@redlink.co
>w: http://redlink.co

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Sergio Fernández]

+1 (binding)

So far I've successfully checked:
* signatures and digests
* source releases file layouts
* matched git tags and commit ids
* incubator suffix and disclaimer
* NOTICE and LICENSE files
* license headers
* clean build (Java 1.8.0_91, Scala, 2.11.7, SBT 0.13.9, Debian amd64)


Couple of minor issues I've seen it'd be great to have fixed in upcoming
releases:
* MongoDbIOTest fails (addr already in use) when a Mongo service is locally
running. I'd say the port should be random in the test suite.
* How did you generated the checksums? Because both SHA1/MD5 can't be
automatically checked because "no properly formatted SHA1/MD5 checksum
lines found".

Great to see the project moving forward at this speed :-)

Cheers,



On Mon, Oct 24, 2016 at 11:30 PM, Aljoscha Krettek 
wrote:

> Hi Team!
>
> Please review and vote at your leisure on release candidate #1 for version
> 0.3.0-incubating, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2],
> * all artifacts to be deployed to the Maven Central Repository [3],
> * source code tag "v0.3.0-incubating-RC1" [4],
> * website pull request listing the release and publishing the API reference
> manual [5].
>
> Please keep in mind that this release is not focused on providing new
> functionality. We want to refine the release process and make stable source
> and binary artefacts available to our users.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PPMC affirmative votes.
>
> Cheers,
> Aljoscha
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527=12338051
> [2] https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-
> incubating/
> [3]
> https://repository.apache.org/content/repositories/staging/
> org/apache/beam/
> [4]
> https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=
> 5d86ff7f04862444c266142b0d5acecb5a6b7144
> [5] https://github.com/apache/incubator-beam-site/pull/52
>



-- 
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernan...@redlink.co
w: http://redlink.co

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Jean-Baptiste Onofré]

It's actually mentionned on the incubator thread usually.

Regards
JB

⁣​

On Oct 25, 2016, 09:46, at 09:46, "Ismaël Mejía"  wrote:
>​+1 (non-binding)
>
>- verified signatures + checksums
>- run mvn clean verify, all artifacts+tests run successfully
>
>One thing to add for future vote calls [idea copied from the latest
>flink
>vote ;)]:
>
>The release artifacts are signed with the key with fingerprint 121D7293
>https://dist.apache.org/repos/dist/release/incubator/beam/KEYS
>​
>
>
>On Tue, Oct 25, 2016 at 7:55 AM, Jean-Baptiste Onofré 
>wrote:
>
>> +1 (binding)
>>
>> I did a quick review and tests. It looks good to me.
>>
>> Regards
>> JB
>>
>> ⁣​
>>
>> On Oct 24, 2016, 23:30, at 23:30, Aljoscha Krettek
>
>> wrote:
>> >Hi Team!
>> >
>> >Please review and vote at your leisure on release candidate #1 for
>> >version
>> >0.3.0-incubating, as follows:
>> >[ ] +1, Approve the release
>> >[ ] -1, Do not approve the release (please provide specific
>comments)
>> >
>> >The complete staging area is available for your review, which
>includes:
>> >* JIRA release notes [1],
>> >* the official Apache source release to be deployed to
>dist.apache.org
>> >[2],
>> >* all artifacts to be deployed to the Maven Central Repository [3],
>> >* source code tag "v0.3.0-incubating-RC1" [4],
>> >* website pull request listing the release and publishing the API
>> >reference
>> >manual [5].
>> >
>> >Please keep in mind that this release is not focused on providing
>new
>> >functionality. We want to refine the release process and make stable
>> >source
>> >and binary artefacts available to our users.
>> >
>> >The vote will be open for at least 72 hours. It is adopted by
>majority
>> >approval, with at least 3 PPMC affirmative votes.
>> >
>> >Cheers,
>> >Aljoscha
>> >
>> >[1]
>> >https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12319527=12338051
>> >[2]
>>
>>https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-incubating/
>> >[3]
>> >https://repository.apache.org/content/repositories/
>> staging/org/apache/beam/
>> >[4]
>>
>>https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=
>> 5d86ff7f04862444c266142b0d5acecb5a6b7144
>> >[5] https://github.com/apache/incubator-beam-site/pull/52
>>

Re: [VOTE] Release 0.3.0-incubating, release candidate #1

[Ismaël Mejía]

​+1 (non-binding)

- verified signatures + checksums
- run mvn clean verify, all artifacts+tests run successfully

One thing to add for future vote calls [idea copied from the latest flink
vote ;)]:

The release artifacts are signed with the key with fingerprint 121D7293
https://dist.apache.org/repos/dist/release/incubator/beam/KEYS
​


On Tue, Oct 25, 2016 at 7:55 AM, Jean-Baptiste Onofré 
wrote:

> +1 (binding)
>
> I did a quick review and tests. It looks good to me.
>
> Regards
> JB
>
> ⁣​
>
> On Oct 24, 2016, 23:30, at 23:30, Aljoscha Krettek 
> wrote:
> >Hi Team!
> >
> >Please review and vote at your leisure on release candidate #1 for
> >version
> >0.3.0-incubating, as follows:
> >[ ] +1, Approve the release
> >[ ] -1, Do not approve the release (please provide specific comments)
> >
> >The complete staging area is available for your review, which includes:
> >* JIRA release notes [1],
> >* the official Apache source release to be deployed to dist.apache.org
> >[2],
> >* all artifacts to be deployed to the Maven Central Repository [3],
> >* source code tag "v0.3.0-incubating-RC1" [4],
> >* website pull request listing the release and publishing the API
> >reference
> >manual [5].
> >
> >Please keep in mind that this release is not focused on providing new
> >functionality. We want to refine the release process and make stable
> >source
> >and binary artefacts available to our users.
> >
> >The vote will be open for at least 72 hours. It is adopted by majority
> >approval, with at least 3 PPMC affirmative votes.
> >
> >Cheers,
> >Aljoscha
> >
> >[1]
> >https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527=12338051
> >[2]
> >https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-incubating/
> >[3]
> >https://repository.apache.org/content/repositories/
> staging/org/apache/beam/
> >[4]
> >https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=
> 5d86ff7f04862444c266142b0d5acecb5a6b7144
> >[5] https://github.com/apache/incubator-beam-site/pull/52
>

Re: [DISCUSS] Current ongoing work on runners

[Jean-Baptiste Onofré]

+1. For me it's one of the most important point for the new website. We should 
give a clear and exhaustive list of what we have, both for runners and IOs 
(with supported features).

Regards
JB

⁣​

On Oct 24, 2016, 21:52, at 21:52, "Ismaël Mejía"  wrote:
>Hello,
>
>I am really happy to see new runners been contributed to our community
>(e.g. GearPump and Apex recently). We have not discussed a lot about
>the
>current capabilities of both runners.
>
>Following the recent discussion about making ongoing work more explicit
>in
>the mailing list, I would like to ask the people involved about the
>current
>status of them, I think it is important to discuss this (apart of
>creating
>the given JIRAs + updating the capability matrix docs) because more
>people
>can eventually jump and give a hand on open issues.
>
>I remember there was a google doc for the  capabilities of each runner,
>is
>this doc still available (sorry I lost the link). I suppose that once
>these
>ongoing runners mature we can add this doc also to the website.
>https://beam.apache.org/learn/runners/capability-matrix/
>
>Regards,
>Ismaël
>
>ps. @Amit, given that the spark 2 (Dataset based) runner has also a
>feature
>branch, if you consider it worth, can you please share a bit about that
>work too.
>
>ps2. Can anyone please share the link to the google doc I was talking
>about, I can't find it after the recent changes to the website.
>​