Re: Enforce javadoc comments in public methods?

[Reuven Lax]

The problem is existing code. I find that if I touch existing code that has
a constructor, I am then forced to add a Javadoc on the constructor.
Usually this Javadoc is "Constructs XX object."

On Mon, Jan 28, 2019 at 5:07 PM Kenneth Knowles  wrote:

> Half agree, half disagree
>
> Disagree: best practice is to make your constructors private and have
> distinct static methods for various modes of instantiation, which will then
> require Javadoc.
>
> Agree: once they are private no javadoc is needed :-) and there's often
> only one "most general" constructor that all the static methods use in
> different ways
>
> Kenn
>
> On Mon, Jan 28, 2019 at 5:03 PM Ruoyun Huang  wrote:
>
>> Fair point. Looking at JavaDocMethod spec [1], unfortunately there is no
>> properties available for this tweak.
>>
>> Let me dig a bit more to see whether this can be done via suppression.
>>
>> [1] http://checkstyle.sourceforge.net/config_javadoc.html#JavadocMethod
>>
>> On Mon, Jan 28, 2019 at 4:36 PM Reuven Lax  wrote:
>>
>>> This appears to be forcing us to set javadoc on constructors as well,
>>> which is usually pointless. Can we exclude constructor methods from this
>>> check?
>>>
>>> On Wed, Jan 23, 2019 at 5:40 PM Ruoyun Huang  wrote:
>>>
 Our recent change is on "JavaDocMethod", which not turned on yet. Not
 relevant to this error here.

 The one throws error is "javaDocType". it has been there for a while
 ,
 which is for public class javadoc missing.  Yeah, I am curious as well why
 preCommit didn't catch this one.



 On Wed, Jan 23, 2019 at 5:28 PM Alex Amato  wrote:

> Did their happen to be a short time window where some missing Javadoc
> comments went in? I am now seeing precommit fail due to code I didn't
> modify.
>
>
> https://scans.gradle.com/s/nwgb7xegklwqo/console-log?task=:beam-runners-direct-java:checkstyleMain
>
>
>
> On Wed, Jan 23, 2019 at 2:34 PM Ruoyun Huang 
> wrote:
>
>> Trying to understand your suggestion. By saying "break that
>> dependency", do you mean moving checkstyle out of Java PreCommit?
>>
>> currently we do have checkstyle as part of  ":check".  It seems to me
>> "check" does minimal amount of essential works (correct me If I am 
>> wrong),
>> much less than what PreCommit does.
>>
>> On Wed, Jan 23, 2019 at 12:20 PM Kenneth Knowles 
>> wrote:
>>
>>> It is always a bummer when the Java PreCommit fails due to style
>>> checking. Can we get this to run separately and quicker? I notice it
>>> depends on compileJava. I cannot remember why that is, but I recall it 
>>> is a
>>> legitimate reason. Nonetheless, can we break that dependency somehow?
>>>
>>> Kenn
>>>
>>> On Wed, Jan 16, 2019 at 6:42 PM Ruoyun Huang 
>>> wrote:
>>>
 Hi, everyone,


 To make sure we move forward to a clean state where we catch
 violations in any new PR, we created this change:
 https://github.com/apache/beam/pull/7532

 This PR makes checkstyle to report error on missing javadocs. For
 existing violations, we explicitly added them as suppression rules, 
 down to
 which line in the code.

 The caveat is, once this PR is merged, whoever make update to any
 file in the list, will very likely have to fix the existing violation 
 for
 that file.  :-) Hope this sounds like a reasonable idea to move 
 forward.

 In the meanwhile, I will try to address the items in the list (if I
 can). And over time, I will get back to this and remove those 
 suppressions
 no longer needed (created JIRA-6446 for tracking purpose), until
 all of them are fixed.

 On Wed, Jan 9, 2019 at 10:57 PM Ruoyun Huang 
 wrote:

> created a PR: https://github.com/apache/beam/pull/7454
>
> Note instead of having separated checkstyle specs for Main versus
> Test, this PR simply uses suppression to turn off JavaDocComment for 
> test
> files.
>
> If this PR draft looks good, then next step I will commit another
> change that:
> 1) throw error on violations (now just warning to keep PR green).
> 2) List all the violations explicitly in a suppression list, and
> let area contributors/owners address and chop things off the list over
> time.  Not ideal and quite some manual work, if there is a better way,
> please let me know.
>
> On Wed, Jan 9, 2019 at 7:29 AM Robert Bradshaw <
> rober...@google.com> wrote:
>
>> On Tue, Jan 8, 2019 at 11:15 PM Kenneth Knowles 
>> wrote:
>> >
>> > I 

Re: [VOTE] Release 2.10.0, release candidate #1

[Kenneth Knowles]

Ah, I did not close the staging repository. Thanks for letting me know. Try
now.

Kenn

On Mon, Jan 28, 2019 at 2:31 PM Ismaël Mejía  wrote:

> I think there is an issue, [4] does not open?
>
> On Mon, Jan 28, 2019 at 6:24 PM Kenneth Knowles  wrote:
> >
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version
> 2.10.0, as follows:
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> [2], which is signed with the key with fingerprint 6ED551A8AE02461C [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.10.0-RC1" [5],
> > * website pull request listing the release [6] and publishing the API
> reference manual [7].
> > * Python artifacts are deployed along with the source release to the
> dist.apache.org [2].
> > * Validation sheet with a tab for 2.10.0 release to help with validation
> [7].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Kenn
> >
> > [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12344540
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.10.0/
> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1056/
> > [5] https://github.com/apache/beam/tree/v2.10.0-RC1
> > [6] https://github.com/apache/beam/pull/7651/files
> > [7] https://github.com/apache/beam-site/pull/585
> > [8]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=2053422529
>

Re: Enforce javadoc comments in public methods?

[Kenneth Knowles]

Half agree, half disagree

Disagree: best practice is to make your constructors private and have
distinct static methods for various modes of instantiation, which will then
require Javadoc.

Agree: once they are private no javadoc is needed :-) and there's often
only one "most general" constructor that all the static methods use in
different ways

Kenn

On Mon, Jan 28, 2019 at 5:03 PM Ruoyun Huang  wrote:

> Fair point. Looking at JavaDocMethod spec [1], unfortunately there is no
> properties available for this tweak.
>
> Let me dig a bit more to see whether this can be done via suppression.
>
> [1] http://checkstyle.sourceforge.net/config_javadoc.html#JavadocMethod
>
> On Mon, Jan 28, 2019 at 4:36 PM Reuven Lax  wrote:
>
>> This appears to be forcing us to set javadoc on constructors as well,
>> which is usually pointless. Can we exclude constructor methods from this
>> check?
>>
>> On Wed, Jan 23, 2019 at 5:40 PM Ruoyun Huang  wrote:
>>
>>> Our recent change is on "JavaDocMethod", which not turned on yet. Not
>>> relevant to this error here.
>>>
>>> The one throws error is "javaDocType". it has been there for a while
>>> ,
>>> which is for public class javadoc missing.  Yeah, I am curious as well why
>>> preCommit didn't catch this one.
>>>
>>>
>>>
>>> On Wed, Jan 23, 2019 at 5:28 PM Alex Amato  wrote:
>>>
 Did their happen to be a short time window where some missing Javadoc
 comments went in? I am now seeing precommit fail due to code I didn't
 modify.


 https://scans.gradle.com/s/nwgb7xegklwqo/console-log?task=:beam-runners-direct-java:checkstyleMain



 On Wed, Jan 23, 2019 at 2:34 PM Ruoyun Huang  wrote:

> Trying to understand your suggestion. By saying "break that
> dependency", do you mean moving checkstyle out of Java PreCommit?
>
> currently we do have checkstyle as part of  ":check".  It seems to me
> "check" does minimal amount of essential works (correct me If I am wrong),
> much less than what PreCommit does.
>
> On Wed, Jan 23, 2019 at 12:20 PM Kenneth Knowles 
> wrote:
>
>> It is always a bummer when the Java PreCommit fails due to style
>> checking. Can we get this to run separately and quicker? I notice it
>> depends on compileJava. I cannot remember why that is, but I recall it 
>> is a
>> legitimate reason. Nonetheless, can we break that dependency somehow?
>>
>> Kenn
>>
>> On Wed, Jan 16, 2019 at 6:42 PM Ruoyun Huang 
>> wrote:
>>
>>> Hi, everyone,
>>>
>>>
>>> To make sure we move forward to a clean state where we catch
>>> violations in any new PR, we created this change:
>>> https://github.com/apache/beam/pull/7532
>>>
>>> This PR makes checkstyle to report error on missing javadocs. For
>>> existing violations, we explicitly added them as suppression rules, 
>>> down to
>>> which line in the code.
>>>
>>> The caveat is, once this PR is merged, whoever make update to any
>>> file in the list, will very likely have to fix the existing violation 
>>> for
>>> that file.  :-) Hope this sounds like a reasonable idea to move forward.
>>>
>>> In the meanwhile, I will try to address the items in the list (if I
>>> can). And over time, I will get back to this and remove those 
>>> suppressions
>>> no longer needed (created JIRA-6446 for tracking purpose), until
>>> all of them are fixed.
>>>
>>> On Wed, Jan 9, 2019 at 10:57 PM Ruoyun Huang 
>>> wrote:
>>>
 created a PR: https://github.com/apache/beam/pull/7454

 Note instead of having separated checkstyle specs for Main versus
 Test, this PR simply uses suppression to turn off JavaDocComment for 
 test
 files.

 If this PR draft looks good, then next step I will commit another
 change that:
 1) throw error on violations (now just warning to keep PR green).
 2) List all the violations explicitly in a suppression list, and
 let area contributors/owners address and chop things off the list over
 time.  Not ideal and quite some manual work, if there is a better way,
 please let me know.

 On Wed, Jan 9, 2019 at 7:29 AM Robert Bradshaw 
 wrote:

> On Tue, Jan 8, 2019 at 11:15 PM Kenneth Knowles 
> wrote:
> >
> > I think @Internal would be a reasonable annotation to exempt
> from documentation, as that means it is explicitly *not* part of the 
> actual
> public API, as Ismaël alluded to.
>
> We'll probably want a distinct annotation from that. Forced
> comments,
> especially forced-by-an-impartial-metric ones, are often lower
> quality. This is the kind of signal that 

Re: Enforce javadoc comments in public methods?

[Ruoyun Huang]

Fair point. Looking at JavaDocMethod spec [1], unfortunately there is no
properties available for this tweak.

Let me dig a bit more to see whether this can be done via suppression.

[1] http://checkstyle.sourceforge.net/config_javadoc.html#JavadocMethod

On Mon, Jan 28, 2019 at 4:36 PM Reuven Lax  wrote:

> This appears to be forcing us to set javadoc on constructors as well,
> which is usually pointless. Can we exclude constructor methods from this
> check?
>
> On Wed, Jan 23, 2019 at 5:40 PM Ruoyun Huang  wrote:
>
>> Our recent change is on "JavaDocMethod", which not turned on yet. Not
>> relevant to this error here.
>>
>> The one throws error is "javaDocType". it has been there for a while
>> ,
>> which is for public class javadoc missing.  Yeah, I am curious as well why
>> preCommit didn't catch this one.
>>
>>
>>
>> On Wed, Jan 23, 2019 at 5:28 PM Alex Amato  wrote:
>>
>>> Did their happen to be a short time window where some missing Javadoc
>>> comments went in? I am now seeing precommit fail due to code I didn't
>>> modify.
>>>
>>>
>>> https://scans.gradle.com/s/nwgb7xegklwqo/console-log?task=:beam-runners-direct-java:checkstyleMain
>>>
>>>
>>>
>>> On Wed, Jan 23, 2019 at 2:34 PM Ruoyun Huang  wrote:
>>>
 Trying to understand your suggestion. By saying "break that
 dependency", do you mean moving checkstyle out of Java PreCommit?

 currently we do have checkstyle as part of  ":check".  It seems to me
 "check" does minimal amount of essential works (correct me If I am wrong),
 much less than what PreCommit does.

 On Wed, Jan 23, 2019 at 12:20 PM Kenneth Knowles 
 wrote:

> It is always a bummer when the Java PreCommit fails due to style
> checking. Can we get this to run separately and quicker? I notice it
> depends on compileJava. I cannot remember why that is, but I recall it is 
> a
> legitimate reason. Nonetheless, can we break that dependency somehow?
>
> Kenn
>
> On Wed, Jan 16, 2019 at 6:42 PM Ruoyun Huang 
> wrote:
>
>> Hi, everyone,
>>
>>
>> To make sure we move forward to a clean state where we catch
>> violations in any new PR, we created this change:
>> https://github.com/apache/beam/pull/7532
>>
>> This PR makes checkstyle to report error on missing javadocs. For
>> existing violations, we explicitly added them as suppression rules, down 
>> to
>> which line in the code.
>>
>> The caveat is, once this PR is merged, whoever make update to any
>> file in the list, will very likely have to fix the existing violation for
>> that file.  :-) Hope this sounds like a reasonable idea to move forward.
>>
>> In the meanwhile, I will try to address the items in the list (if I
>> can). And over time, I will get back to this and remove those 
>> suppressions
>> no longer needed (created JIRA-6446 for tracking purpose), until all
>> of them are fixed.
>>
>> On Wed, Jan 9, 2019 at 10:57 PM Ruoyun Huang 
>> wrote:
>>
>>> created a PR: https://github.com/apache/beam/pull/7454
>>>
>>> Note instead of having separated checkstyle specs for Main versus
>>> Test, this PR simply uses suppression to turn off JavaDocComment for 
>>> test
>>> files.
>>>
>>> If this PR draft looks good, then next step I will commit another
>>> change that:
>>> 1) throw error on violations (now just warning to keep PR green).
>>> 2) List all the violations explicitly in a suppression list, and let
>>> area contributors/owners address and chop things off the list over time.
>>> Not ideal and quite some manual work, if there is a better way, please 
>>> let
>>> me know.
>>>
>>> On Wed, Jan 9, 2019 at 7:29 AM Robert Bradshaw 
>>> wrote:
>>>
 On Tue, Jan 8, 2019 at 11:15 PM Kenneth Knowles 
 wrote:
 >
 > I think @Internal would be a reasonable annotation to exempt from
 documentation, as that means it is explicitly *not* part of the actual
 public API, as Ismaël alluded to.

 We'll probably want a distinct annotation from that. Forced
 comments,
 especially forced-by-an-impartial-metric ones, are often lower
 quality. This is the kind of signal that would be useful to surface
 to
 a reviewer who could then (jointly) make the call rather than it
 being
 a binary failure/success.

 > (I'm still on the docs-on-private-too side of things, but realize
 that's an extreme position)

 +1 to docs on private things as well, though maybe with not as high
 priority :).

 > It is a shame that we chose blacklist (via @Internal) instead of
 whitelist (via e.g. @Public) for what constitutes an 

Re: Enforce javadoc comments in public methods?

[Reuven Lax]

This appears to be forcing us to set javadoc on constructors as well, which
is usually pointless. Can we exclude constructor methods from this check?

On Wed, Jan 23, 2019 at 5:40 PM Ruoyun Huang  wrote:

> Our recent change is on "JavaDocMethod", which not turned on yet. Not
> relevant to this error here.
>
> The one throws error is "javaDocType". it has been there for a while
> ,
> which is for public class javadoc missing.  Yeah, I am curious as well why
> preCommit didn't catch this one.
>
>
>
> On Wed, Jan 23, 2019 at 5:28 PM Alex Amato  wrote:
>
>> Did their happen to be a short time window where some missing Javadoc
>> comments went in? I am now seeing precommit fail due to code I didn't
>> modify.
>>
>>
>> https://scans.gradle.com/s/nwgb7xegklwqo/console-log?task=:beam-runners-direct-java:checkstyleMain
>>
>>
>>
>> On Wed, Jan 23, 2019 at 2:34 PM Ruoyun Huang  wrote:
>>
>>> Trying to understand your suggestion. By saying "break that dependency",
>>> do you mean moving checkstyle out of Java PreCommit?
>>>
>>> currently we do have checkstyle as part of  ":check".  It seems to me
>>> "check" does minimal amount of essential works (correct me If I am wrong),
>>> much less than what PreCommit does.
>>>
>>> On Wed, Jan 23, 2019 at 12:20 PM Kenneth Knowles 
>>> wrote:
>>>
 It is always a bummer when the Java PreCommit fails due to style
 checking. Can we get this to run separately and quicker? I notice it
 depends on compileJava. I cannot remember why that is, but I recall it is a
 legitimate reason. Nonetheless, can we break that dependency somehow?

 Kenn

 On Wed, Jan 16, 2019 at 6:42 PM Ruoyun Huang  wrote:

> Hi, everyone,
>
>
> To make sure we move forward to a clean state where we catch
> violations in any new PR, we created this change:
> https://github.com/apache/beam/pull/7532
>
> This PR makes checkstyle to report error on missing javadocs. For
> existing violations, we explicitly added them as suppression rules, down 
> to
> which line in the code.
>
> The caveat is, once this PR is merged, whoever make update to any file
> in the list, will very likely have to fix the existing violation for that
> file.  :-) Hope this sounds like a reasonable idea to move forward.
>
> In the meanwhile, I will try to address the items in the list (if I
> can). And over time, I will get back to this and remove those suppressions
> no longer needed (created JIRA-6446 for tracking purpose), until all
> of them are fixed.
>
> On Wed, Jan 9, 2019 at 10:57 PM Ruoyun Huang 
> wrote:
>
>> created a PR: https://github.com/apache/beam/pull/7454
>>
>> Note instead of having separated checkstyle specs for Main versus
>> Test, this PR simply uses suppression to turn off JavaDocComment for test
>> files.
>>
>> If this PR draft looks good, then next step I will commit another
>> change that:
>> 1) throw error on violations (now just warning to keep PR green).
>> 2) List all the violations explicitly in a suppression list, and let
>> area contributors/owners address and chop things off the list over time.
>> Not ideal and quite some manual work, if there is a better way, please 
>> let
>> me know.
>>
>> On Wed, Jan 9, 2019 at 7:29 AM Robert Bradshaw 
>> wrote:
>>
>>> On Tue, Jan 8, 2019 at 11:15 PM Kenneth Knowles 
>>> wrote:
>>> >
>>> > I think @Internal would be a reasonable annotation to exempt from
>>> documentation, as that means it is explicitly *not* part of the actual
>>> public API, as Ismaël alluded to.
>>>
>>> We'll probably want a distinct annotation from that. Forced comments,
>>> especially forced-by-an-impartial-metric ones, are often lower
>>> quality. This is the kind of signal that would be useful to surface
>>> to
>>> a reviewer who could then (jointly) make the call rather than it
>>> being
>>> a binary failure/success.
>>>
>>> > (I'm still on the docs-on-private-too side of things, but realize
>>> that's an extreme position)
>>>
>>> +1 to docs on private things as well, though maybe with not as high
>>> priority :).
>>>
>>> > It is a shame that we chose blacklist (via @Internal) instead of
>>> whitelist (via e.g. @Public) for what constitutes an actual supported
>>> public method.
>>>
>>> Probably better than having to re-train others that public doesn't
>>> really mean public unless it has a @Public on it. It's harder to
>>> "unknowingly" use an @Internal API.
>>>
>>>
>>> > Kenn
>>> >
>>> > On Tue, Jan 8, 2019 at 1:46 PM Ruoyun Huang 
>>> wrote:
>>> >>
>>> >> To Ismael's question:  When applying such a check (i.e. public
>>> method with >30 Loc), 

Re: [VOTE] Release 2.10.0, release candidate #1

[Ismaël Mejía]

I think there is an issue, [4] does not open?

On Mon, Jan 28, 2019 at 6:24 PM Kenneth Knowles  wrote:
>
> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 2.10.0, as 
> follows:
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org [2], 
> which is signed with the key with fingerprint 6ED551A8AE02461C [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.10.0-RC1" [5],
> * website pull request listing the release [6] and publishing the API 
> reference manual [7].
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org [2].
> * Validation sheet with a tab for 2.10.0 release to help with validation [7].
>
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Kenn
>
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12344540
> [2] https://dist.apache.org/repos/dist/dev/beam/2.10.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1056/
> [5] https://github.com/apache/beam/tree/v2.10.0-RC1
> [6] https://github.com/apache/beam/pull/7651/files
> [7] https://github.com/apache/beam-site/pull/585
> [8] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=2053422529

Re: compileJava broken on master see: BEAM-6495

[Alex Amato]

If it continues to occur, maybe it is an environmental issue, be sure to
try to clean as well.
./gradlew clean

On Mon, Jan 28, 2019 at 11:12 AM Ryan Williams  wrote:

> Yea I was rebased on top of a more recent master than your previous
> message, when I saw it again. Perhaps I was mistaken. I'll ping here if I
> see it again, thanks.
>
> On Mon, Jan 28, 2019 at 1:52 PM Alex Amato  wrote:
>
>> After I did a rebase, it went away for me. So I think that this should
>> work. Are you saying that you did rebase ontop of master and it still
>> occurred? Strange.
>>
>> On Sat, Jan 26, 2019 at 3:48 PM Ryan Williams 
>> wrote:
>>
>>> Hm, I just encountered this again on a branch that based on 5b46b02b49
>>> (top of trunk from this afternoon). Is it definitely supposed to be fixed?
>>>
>>>
>>> On Thu, Jan 24, 2019 at 9:19 PM Alex Amato  wrote:
>>>
 Please try rebasing from master, I believe this issue has been resolved.

 On Thu, Jan 24, 2019 at 3:29 PM Ryan Williams 
 wrote:

> I'm seeing every ≈third `./gradlew compileJava` fail locally due to
> this; re-running the commit has always succeeded, so far.
>
> Sounds like there is not an immediate fix in the works / no one
> assigned on the JIRA?
>
> On Wed, Jan 23, 2019 at 3:17 PM Kenneth Knowles 
> wrote:
>
>> This might connect to vendoring Calcite. It will be easiest, and have
>> the best incremental build, if we separate the generated code into its 
>> own
>> module that has relocation to match the vendored Calcite.
>>
>> Kenn
>>
>> On Wed, Jan 23, 2019 at 11:29 AM Anton Kedin 
>> wrote:
>>
>>> We don't pre-generate the code as a separate step. Code gen from the
>>> SQL parser syntax spec and its compilation happens both during the Beam 
>>> SQL
>>> build task. Splitting the code generation and compilation might not be
>>> trivial. We definitely should look into fixing this though.
>>>
>>> Regards,
>>> Anton
>>>
>>> On Wed, Jan 23, 2019 at 11:13 AM Alex Amato 
>>> wrote:
>>>
 Okay, make sense perhaps we can somehow make it fail when it fails
 to generate the dep, rather than when compiling the java code later on

 On Wed, Jan 23, 2019 at 11:12 AM Anton Kedin 
 wrote:

> ParserImpl is autogenerated by Calcite at build time. It seems
> that there's a race condition there and it sometimes fails. Rerunning 
> the
> build works for me.
>
> Regards,
> Anton
>
> On Wed, Jan 23, 2019, 11:06 AM Alex Amato 
> wrote:
>
>> https://jira.apache.org/jira/browse/BEAM-6495?filter=-2
>>
>> Any ideas, how this got through the precommit?
>>
>> > Task :beam-sdks-java-extensions-sql:compileJava FAILED
>>
>> /usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29:
>> error: cannot find symbol
>>
>> import
>> org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl;
>>
>>   ^
>>
>>   symbol:   class BeamSqlParserImpl
>>
>>   location: package
>> org.apache.beam.sdk.extensions.sql.impl.parser.impl
>>
>> 1 error
>>
>>

Re: Beam Dependency Check Report (2018-06-13)

[Ahmet Altay]

Looking at the latest report for January 28, there are lots of stale
dependencies. Their associated JIRAs are open for more than 3 months in
some cases. We have a decent policy and tooling to support that, but we are
not following up on those with actions. How could we keep the identified
stale dependencies up to date? Could we mark the issue with a fix version
and triage during releases? That will allow us to at least sort out major
dependency updates from the less urgent ones.

On Mon, Jan 28, 2019 at 9:43 AM Yifan Zou  wrote:

> Hi,
>
> You're looking at the old versions dependency bugs which were created
> before Oct, 2018 (e.g BEAM-4904
> ). Based on the
> discussion [1]
> ,
> we modified the tool with the new Beam Dependency Policy
> , and closed the old
> bugs (most of them were marked as won't fix, and they will never get
> updated).
>
> The current dependency JIRA looks like this: BEAM-5549
> . The major changes
> including [2] :
>
> 1. A JIRA will be created if a dependency has more then 1 major version or
> 3 minor versions behind the latest version. Or, there is new version
> available for more then a year that the dep didn't update in Beam.
> 2. A JIRA could be closed if the new version is not appropriate to be used
> in Beam. In this case, the tool will stop checking updates on this dep
> until the next major version available or after 3 months.
> 3. Stop specifying the target version number in the issue's title. This
> ensures that only one JIRA would be opened for a dep that people can easily
> track the update history.
> 4. Stop directly assigning bugs to a person. Instead, cc owners in the
> descriptions.
>
> Please use the new dependency JIRAs to track the updates. Thanks for
> taking care of Beam dependencies and let me know if you have any questions
> and concerns.
>
> Regards.
> Yifan
>
> [1]:
> https://lists.apache.org/thread.html/28d3c349a5021c3598379b6f6b9210b4ef150a6235e55c0499250034@%3Cdev.beam.apache.org%3E
> [2]: https://issues.apache.org/jira/browse/BEAM-5339
>
> On Mon, Jan 28, 2019 at 6:22 AM Ismaël Mejía  wrote:
>
>> Hello,
>>
>> The dependency update report has been working fine. However I found some
>> issues that I summarized in this issue.
>> https://issues.apache.org/jira/browse/BEAM-6524
>> Can Yifan or someone else that knows that area please take a look.
>>
>> Regards,
>> Ismaël
>>
>>
>> On Thu, Jun 14, 2018 at 11:37 PM Yifan Zou  wrote:
>>
>>> Thank you Paul for letting us know this issue. We will take care of it
>>> when upgrading dependencies.
>>>
>>> On Thu, Jun 14, 2018 at 7:23 AM Paul Gerver  wrote:
>>>
 I do have one request to be added to the Java SDK version updates:
 Beam-3831 [1]. The Google Core depends on the old org.json package which
 ASF discourages using because of the "Use only for good, not evil" clause.

 [1] https://issues.apache.org/jira/browse/BEAM-3831

 On Thu, Jun 14, 2018 at 3:03 AM Etienne Chauchot 
 wrote:

> Thanks Yifan,
>
> This is great ! It would help us maintain Beam more easily and
> probably help us fixing CVE as well.
>
> Etienne
>
> Le mercredi 13 juin 2018 à 07:45 -0700, Yifan Zou a écrit :
>
> Hi,
>
>
> I want to follow up and explain this email.
>
>
> This is a sample email that reports the results of Beam SDK dependency
> check, which was proposed here
> .
> The goal is finding updates for all Beam Python & Java SDKs' dependencies
> and prioritize them. The job will be auto triggered in Jenkins once a week
> and generate a report. The report lists the high priority updates base on
> the following criteria:
>
>
> The dependency update is high priority if:
>
> 1. It has major versions update available;
>
>   e.g. org.assertj:assertj-core 2.5.0 -> 3.10.0
>
>  2. or, it is over 3 minor versions behind the latest version;
>
>   e.g. org.tukaani:xz 1.5 -> 1.8
>
> 3. or, the current version is behind the later version for over 180
> days.
>
>   e.g. com.google.auto.service:auto-service 2014-10-24 ->
> 2017-12-11
>
>
> This job helps Beam contributors to determine the dependency which is
> far behind the latest released version. The next step would be automating
> filing JIRA bugs for dep updates, group dependencies and identify owners 
> to
> take care of the upgrades follow Chamikara's proposal
> 

Re: compileJava broken on master see: BEAM-6495

[Ryan Williams]

Yea I was rebased on top of a more recent master than your previous
message, when I saw it again. Perhaps I was mistaken. I'll ping here if I
see it again, thanks.

On Mon, Jan 28, 2019 at 1:52 PM Alex Amato  wrote:

> After I did a rebase, it went away for me. So I think that this should
> work. Are you saying that you did rebase ontop of master and it still
> occurred? Strange.
>
> On Sat, Jan 26, 2019 at 3:48 PM Ryan Williams 
> wrote:
>
>> Hm, I just encountered this again on a branch that based on 5b46b02b49
>> (top of trunk from this afternoon). Is it definitely supposed to be fixed?
>>
>>
>> On Thu, Jan 24, 2019 at 9:19 PM Alex Amato  wrote:
>>
>>> Please try rebasing from master, I believe this issue has been resolved.
>>>
>>> On Thu, Jan 24, 2019 at 3:29 PM Ryan Williams 
>>> wrote:
>>>
 I'm seeing every ≈third `./gradlew compileJava` fail locally due to
 this; re-running the commit has always succeeded, so far.

 Sounds like there is not an immediate fix in the works / no one
 assigned on the JIRA?

 On Wed, Jan 23, 2019 at 3:17 PM Kenneth Knowles 
 wrote:

> This might connect to vendoring Calcite. It will be easiest, and have
> the best incremental build, if we separate the generated code into its own
> module that has relocation to match the vendored Calcite.
>
> Kenn
>
> On Wed, Jan 23, 2019 at 11:29 AM Anton Kedin  wrote:
>
>> We don't pre-generate the code as a separate step. Code gen from the
>> SQL parser syntax spec and its compilation happens both during the Beam 
>> SQL
>> build task. Splitting the code generation and compilation might not be
>> trivial. We definitely should look into fixing this though.
>>
>> Regards,
>> Anton
>>
>> On Wed, Jan 23, 2019 at 11:13 AM Alex Amato 
>> wrote:
>>
>>> Okay, make sense perhaps we can somehow make it fail when it fails
>>> to generate the dep, rather than when compiling the java code later on
>>>
>>> On Wed, Jan 23, 2019 at 11:12 AM Anton Kedin 
>>> wrote:
>>>
 ParserImpl is autogenerated by Calcite at build time. It seems that
 there's a race condition there and it sometimes fails. Rerunning the 
 build
 works for me.

 Regards,
 Anton

 On Wed, Jan 23, 2019, 11:06 AM Alex Amato 
 wrote:

> https://jira.apache.org/jira/browse/BEAM-6495?filter=-2
>
> Any ideas, how this got through the precommit?
>
> > Task :beam-sdks-java-extensions-sql:compileJava FAILED
>
> /usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29:
> error: cannot find symbol
>
> import
> org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl;
>
>   ^
>
>   symbol:   class BeamSqlParserImpl
>
>   location: package
> org.apache.beam.sdk.extensions.sql.impl.parser.impl
>
> 1 error
>
>

Re: compileJava broken on master see: BEAM-6495

[Alex Amato]

After I did a rebase, it went away for me. So I think that this should
work. Are you saying that you did rebase ontop of master and it still
occurred? Strange.

On Sat, Jan 26, 2019 at 3:48 PM Ryan Williams  wrote:

> Hm, I just encountered this again on a branch that based on 5b46b02b49
> (top of trunk from this afternoon). Is it definitely supposed to be fixed?
>
>
> On Thu, Jan 24, 2019 at 9:19 PM Alex Amato  wrote:
>
>> Please try rebasing from master, I believe this issue has been resolved.
>>
>> On Thu, Jan 24, 2019 at 3:29 PM Ryan Williams 
>> wrote:
>>
>>> I'm seeing every ≈third `./gradlew compileJava` fail locally due to
>>> this; re-running the commit has always succeeded, so far.
>>>
>>> Sounds like there is not an immediate fix in the works / no one assigned
>>> on the JIRA?
>>>
>>> On Wed, Jan 23, 2019 at 3:17 PM Kenneth Knowles  wrote:
>>>
 This might connect to vendoring Calcite. It will be easiest, and have
 the best incremental build, if we separate the generated code into its own
 module that has relocation to match the vendored Calcite.

 Kenn

 On Wed, Jan 23, 2019 at 11:29 AM Anton Kedin  wrote:

> We don't pre-generate the code as a separate step. Code gen from the
> SQL parser syntax spec and its compilation happens both during the Beam 
> SQL
> build task. Splitting the code generation and compilation might not be
> trivial. We definitely should look into fixing this though.
>
> Regards,
> Anton
>
> On Wed, Jan 23, 2019 at 11:13 AM Alex Amato 
> wrote:
>
>> Okay, make sense perhaps we can somehow make it fail when it fails to
>> generate the dep, rather than when compiling the java code later on
>>
>> On Wed, Jan 23, 2019 at 11:12 AM Anton Kedin 
>> wrote:
>>
>>> ParserImpl is autogenerated by Calcite at build time. It seems that
>>> there's a race condition there and it sometimes fails. Rerunning the 
>>> build
>>> works for me.
>>>
>>> Regards,
>>> Anton
>>>
>>> On Wed, Jan 23, 2019, 11:06 AM Alex Amato 
>>> wrote:
>>>
 https://jira.apache.org/jira/browse/BEAM-6495?filter=-2

 Any ideas, how this got through the precommit?

 > Task :beam-sdks-java-extensions-sql:compileJava FAILED

 /usr/local/google/home/ajamato/go/src/
 github.com/apache/beam/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29:
 error: cannot find symbol

 import
 org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl;

   ^

   symbol:   class BeamSqlParserImpl

   location: package
 org.apache.beam.sdk.extensions.sql.impl.parser.impl

 1 error

Re: The full list of proposals / prototype documents

[Ismaël Mejía]

First step probably is to make mandatory to add the document to the
design documents page before sending it to the mailing list for
discussion.

On Mon, Jan 28, 2019 at 6:13 PM Maximilian Michels  wrote:
>
> This is awesome. Thank you for creating this page Alexey! I found some 
> documents
> which I missed in the past.
>
> I wonder if we could make it easier to maintain the page? If design docs were 
> in
> the Wiki, it would be trivial to maintain such a list. Obviously the Google 
> Docs
> are great for commenting/collaboration, but they are hard to discover.
>
> On 25.01.19 18:37, Alexey Romanenko wrote:
> > Hi Alex,
> >
> > Indeed, some of them are not shared with everyone. I asked on Slack channel 
> > to
> > fix this.
> > Thank you for the list!
> >
> >
> >> On 18 Jan 2019, at 19:57, Alex Van Boxel  >> > wrote:
> >>
> >> typically me... I just click on 2 of the 3 that were not shared. I went 
> >> over
> >> all of the proposals to see where I needed to get access, here is the list:
> >>
> >>
> >>   SQL / Schema
> >>
> >>   * Pubsub to Beam SQL [doc
> >> 
> >> ]
> >>   * Calcite/Beam SQL Windowing [doc
> >> 
> >> ]
> >>   * Reject Unsupported Windowing Strategies in JOIN [doc
> >> 
> >> ]
> >>
> >> the ones that I had access to, from the past I don't know anymore of 
> >> course.
> >>
> >>
> >>
> >>  _/
> >> _/ Alex Van Boxel
> >>
> >>
> >> On Fri, Jan 18, 2019 at 6:13 PM Alexey Romanenko  >> > wrote:
> >>
> >> Hi Alex,
> >>
> >> Hmm, afaik, this is mostly google docs file which shared with anyone 
> >> who
> >> knows the link.
> >> Could you send here the names of proposals that required an access 
> >> approval?
> >> Thanks.
> >>
> >>> On 18 Jan 2019, at 16:58, Alex Van Boxel  >>> > wrote:
> >>>
> >>> Hey Alexey,
> >>>
> >>> I see that a lot (well, I tried 2) proposals require access approval.
> >>> Should that be the case?
> >>>
> >>>  _/
> >>> _/ Alex Van Boxel
> >>>
> >>>
> >>> On Fri, Jan 18, 2019 at 4:51 PM Alexey Romanenko
> >>> mailto:aromanenko@gmail.com>> wrote:
> >>>
> >>> I’m sorry but I forgot to mention that the whole list could be 
> >>> found
> >>> here:
> >>> https://beam.apache.org/contribute/design-documents/
> >>>
> >>>
>  On 18 Jan 2019, at 16:49, Alexey Romanenko 
>    > wrote:
> 
>  FYI: I updated the list of design documents to make it 
>  up-to-date.
>  PR: https://github.com/apache/beam/pull/7560
>  Please, feel free to add new ones if I missed something.
> 
>  Also, I’d like to remind that it would be very helpful to add 
>  design
>  document to this page in the same time as it was created and
>  finalised after all discussions.
>  It should help us to keep this list consistent and always 
>  up-to-date.
>  Thank you in advance!
> 
>  On 11 Jul 2018, at 17:03, Alexey Romanenko 
>    > wrote:
> >
> > Thank you for this link, Etienne.
> > I agree that it doesn’t fit well for design documents page. So, 
> > I
> > think it makes sense to add either on wiki or as a part of 
> > Nexmark
> > documentation on web site:
> > https://beam.apache.org/documentation/sdks/java/nexmark/.
> >
> >
> > On 11 Jul 2018, at 11:16, Etienne Chauchot  > > wrote:
> >>
> >> Alexey,
> >>
> >> One doc that can be interesting that I forgot to point out is
> >> 
> >> https://docs.google.com/document/d/1VgnGiVu8vSfm7Et-xAtQYv0PlEpqeyfmhpQUNPmWRJs/edit?usp=sharing
> >> It is the doc I wrote when I submitted Nexmark PR to ease the
> >> reading of the code.
> >> It is not a design doc, I don't know if it belongs to the 
> >> website
> >> page or to the wiki for beam devs.
> >>
> >> Etienne
> >>
> >>
> >> Le mercredi 06 juin 2018 à 17:48 +0200, Alexey Romanenko a 
> >> écrit :
> >>> FYI: Finally, it was merged and you can find this page here:
> >>> https://beam.apache.org/contribute/design-documents/
> >>>
> >>> Thank you everybody who helped me to compile this list!
> >>> I’ll try to do my best to update this with new coming docs. In
> >>> the same time, please, feel free to add your new docs (or 
> >>> notify
> 

Re: Beam Dependency Check Report (2018-06-13)

[Yifan Zou]

Hi,

You're looking at the old versions dependency bugs which were created
before Oct, 2018 (e.g BEAM-4904
). Based on the discussion
[1]
,
we modified the tool with the new Beam Dependency Policy
, and closed the old bugs
(most of them were marked as won't fix, and they will never get updated).

The current dependency JIRA looks like this: BEAM-5549
. The major changes
including [2] :

1. A JIRA will be created if a dependency has more then 1 major version or
3 minor versions behind the latest version. Or, there is new version
available for more then a year that the dep didn't update in Beam.
2. A JIRA could be closed if the new version is not appropriate to be used
in Beam. In this case, the tool will stop checking updates on this dep
until the next major version available or after 3 months.
3. Stop specifying the target version number in the issue's title. This
ensures that only one JIRA would be opened for a dep that people can easily
track the update history.
4. Stop directly assigning bugs to a person. Instead, cc owners in the
descriptions.

Please use the new dependency JIRAs to track the updates. Thanks for taking
care of Beam dependencies and let me know if you have any questions and
concerns.

Regards.
Yifan

[1]:
https://lists.apache.org/thread.html/28d3c349a5021c3598379b6f6b9210b4ef150a6235e55c0499250034@%3Cdev.beam.apache.org%3E
[2]: https://issues.apache.org/jira/browse/BEAM-5339

On Mon, Jan 28, 2019 at 6:22 AM Ismaël Mejía  wrote:

> Hello,
>
> The dependency update report has been working fine. However I found some
> issues that I summarized in this issue.
> https://issues.apache.org/jira/browse/BEAM-6524
> Can Yifan or someone else that knows that area please take a look.
>
> Regards,
> Ismaël
>
>
> On Thu, Jun 14, 2018 at 11:37 PM Yifan Zou  wrote:
>
>> Thank you Paul for letting us know this issue. We will take care of it
>> when upgrading dependencies.
>>
>> On Thu, Jun 14, 2018 at 7:23 AM Paul Gerver  wrote:
>>
>>> I do have one request to be added to the Java SDK version updates:
>>> Beam-3831 [1]. The Google Core depends on the old org.json package which
>>> ASF discourages using because of the "Use only for good, not evil" clause.
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-3831
>>>
>>> On Thu, Jun 14, 2018 at 3:03 AM Etienne Chauchot 
>>> wrote:
>>>
 Thanks Yifan,

 This is great ! It would help us maintain Beam more easily and probably
 help us fixing CVE as well.

 Etienne

 Le mercredi 13 juin 2018 à 07:45 -0700, Yifan Zou a écrit :

 Hi,


 I want to follow up and explain this email.


 This is a sample email that reports the results of Beam SDK dependency
 check, which was proposed here
 .
 The goal is finding updates for all Beam Python & Java SDKs' dependencies
 and prioritize them. The job will be auto triggered in Jenkins once a week
 and generate a report. The report lists the high priority updates base on
 the following criteria:


 The dependency update is high priority if:

 1. It has major versions update available;

   e.g. org.assertj:assertj-core 2.5.0 -> 3.10.0

  2. or, it is over 3 minor versions behind the latest version;

   e.g. org.tukaani:xz 1.5 -> 1.8

 3. or, the current version is behind the later version for over 180
 days.

   e.g. com.google.auto.service:auto-service 2014-10-24 ->
 2017-12-11


 This job helps Beam contributors to determine the dependency which is
 far behind the latest released version. The next step would be automating
 filing JIRA bugs for dep updates, group dependencies and identify owners to
 take care of the upgrades follow Chamikara's proposal
 
 .


 For more readings:

 [Proposal] Beam dependency check automation
 
  by Yifan Zou

 [Proposal] Beam dependency update policy
 
  by *Chamikara Jayalath*

 Thank you.

 Yifan Zou

 On Wed, Jun 13, 2018 at 7:41 AM Apache Jenkins Server <
 jenk...@builds.apache.org> wrote:

 High Priority Dependency Updates Of Beam Python SDK:
 *Dependency Name* *Current Version* *Later Version* *Current Version

[VOTE] Release 2.10.0, release candidate #1

[Kenneth Knowles]

Hi everyone,

Please review and vote on the release candidate #1 for the version 2.10.0,
as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint 6ED551A8AE02461C [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.10.0-RC1" [5],
* website pull request listing the release [6] and publishing the API
reference manual [7].
* Python artifacts are deployed along with the source release to the
dist.apache.org [2].
* Validation sheet with a tab for 2.10.0 release to help with validation
[7].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Kenn

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12344540
[2] https://dist.apache.org/repos/dist/dev/beam/2.10.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1056/
[5] https://github.com/apache/beam/tree/v2.10.0-RC1
[6] https://github.com/apache/beam/pull/7651/files
[7] https://github.com/apache/beam-site/pull/585
[8]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=2053422529

Re: [Proposal] Requesting PMC approval to start planning for Beam Summits 2019

[Maximilian Michels]

Was this intended to be a private document only for the PMC? I think it would be 
ok to share it with everyone.


@Joana Just pinging in case you didn't see this on the mailing list.

Thanks,
Max

On 26.01.19 17:19, Thomas Weise wrote:

The document isn't accessible, please open for comments.


On Fri, Jan 25, 2019 at 5:29 PM Joana Filipa Bernardo Carrasqueira 
mailto:joanafil...@google.com>> wrote:


Dear Project Management Committee,


Thank you for your feedback. As suggested I willfocus on the organization of
the Beam Summit North America (SF) first and as requested, please see here


a document with more information about the roles of the Organizers and 
Sponsors.


Happy to provide more information to you and the community if needed.


Kind Regards,

Joana


On Sun, Jan 20, 2019 at 9:58 PM Davor Bonaci mailto:da...@apache.org>> wrote:

I'm sorry, but certain things require special care. General organizing
discussions can proceed on dev@, excluding this specific one.

On Sun, Jan 20, 2019 at 8:43 PM Gris Cuevas mailto:g...@apache.org>> wrote:



On 2019/01/19 16:58:52, Davor Bonaci mailto:da...@apache.org>> wrote:
 > I'd say these matters are generally private between the
organizer(s) and
 > the PMC. This thread should continue on the PMC-private mailing 
list.

Last time we checked (back in October), people who are not part of
the PMC couldn't send emails to the private mailing list nor read
the responses. The organization of the last London Summit occurred
in the private list with me and Matthias directly cc'ed in our
individual emails. Additionally a few of the other organizers didn't
get visibility on how decisions were formed and this obstructed the
planning.

I'd suggest we continue with the initial discussion here in order to
offer transparency, if a sensitive matter needs to be discussed, we
can call it out here and set context that a specific conversation
will be moved to private, bringing the decision made back to this
thread.

Also, I'd recommend we keep the discussion focused on the main
matter: Summit planning. If you have a proposal to revise what
communications need to be had in specific channels, you can start
the conversation in a new separate thread, I see a lot of value in
having it.

@PMC members, we'd love to get an outline of what needs to be done
to continue with the planning. We have a few people interested in
helping out and we'd love to keep the momentum going.

 >
 > On Fri, Jan 18, 2019 at 4:06 PM Ahmet Altay mailto:al...@google.com>> wrote:
 >
 > > Thank you Joana.
 > >
 > > Kenn and PMC members could you comment on what needs to be done
to move
 > > this forward?
 > >
 > > On Thu, Jan 17, 2019 at 3:40 PM joanafil...@google.com
 <
 > > joanafil...@google.com > wrote:
 > >
 > >> Dear Project Management Committee,
 > >>
 > >>
 > >> The Beam Summits are community events funded by a Sponsoring
Committee
 > >> and organized by an Organizing Committee. I’d like to get the
following
 > >> approvals:
 > >>
 > >> To organize and host the Summits under the name of Beam Summit

 > >> , i.e. Beam Summit North America 2019, Beam Summit
Europe 2019 and
 > >> Beam Summit Asia 2019.
 > >>
 > >> To form organizing committees for each edition
 > >>
 > >> Approval to host each edition on the following dates and
locations:
 > >>
 > >> - Beam Summit North America, on April 3rd, San Francisco, CA. 
(150
 > >> attendees)
 > >>
 > >> - Beam Summit Europe, on June 19th, Berlin, Germany. (150
attendees)
 > >>
 > >> - Beam Summit Asia, October (exact date tbc), Tokyo, Japan. 
(150
 > >> attendees)
 > >>
 > >> The events will provide educational content selected by the
Organizing
 > >> Committee, and will be a not-for-profit event, however, we
might charge a
 > >> fee to support the organization and logistics costs. This
matter will be
 > >> decided by the Organizing Committee and will be brought back
to the PMC if
  

Re: The full list of proposals / prototype documents

[Maximilian Michels]

This is awesome. Thank you for creating this page Alexey! I found some documents 
which I missed in the past.


I wonder if we could make it easier to maintain the page? If design docs were in 
the Wiki, it would be trivial to maintain such a list. Obviously the Google Docs 
are great for commenting/collaboration, but they are hard to discover.


On 25.01.19 18:37, Alexey Romanenko wrote:

Hi Alex,

Indeed, some of them are not shared with everyone. I asked on Slack channel to 
fix this.

Thank you for the list!


On 18 Jan 2019, at 19:57, Alex Van Boxel > wrote:


typically me... I just click on 2 of the 3 that were not shared. I went over 
all of the proposals to see where I needed to get access, here is the list:



  SQL / Schema

  * Pubsub to Beam SQL [doc

]
  * Calcite/Beam SQL Windowing [doc

]
  * Reject Unsupported Windowing Strategies in JOIN [doc

]

the ones that I had access to, from the past I don't know anymore of course.



 _/
_/ Alex Van Boxel


On Fri, Jan 18, 2019 at 6:13 PM Alexey Romanenko > wrote:


Hi Alex,

Hmm, afaik, this is mostly google docs file which shared with anyone who
knows the link.
Could you send here the names of proposals that required an access approval?
Thanks.


On 18 Jan 2019, at 16:58, Alex Van Boxel mailto:a...@vanboxel.be>> wrote:

Hey Alexey,

I see that a lot (well, I tried 2) proposals require access approval.
Should that be the case?

 _/
_/ Alex Van Boxel


On Fri, Jan 18, 2019 at 4:51 PM Alexey Romanenko
mailto:aromanenko@gmail.com>> wrote:

I’m sorry but I forgot to mention that the whole list could be found
here:
https://beam.apache.org/contribute/design-documents/



On 18 Jan 2019, at 16:49, Alexey Romanenko mailto:aromanenko@gmail.com>> wrote:

FYI: I updated the list of design documents to make it up-to-date.
PR: https://github.com/apache/beam/pull/7560
Please, feel free to add new ones if I missed something.

Also, I’d like to remind that it would be very helpful to add design
document to this page in the same time as it was created and
finalised after all discussions.
It should help us to keep this list consistent and always up-to-date.
Thank you in advance!

On 11 Jul 2018, at 17:03, Alexey Romanenko mailto:aromanenko@gmail.com>> wrote:


Thank you for this link, Etienne.
I agree that it doesn’t fit well for design documents page. So, I
think it makes sense to add either on wiki or as a part of Nexmark
documentation on web site:
https://beam.apache.org/documentation/sdks/java/nexmark/.


On 11 Jul 2018, at 11:16, Etienne Chauchot mailto:echauc...@apache.org>> wrote:


Alexey,

One doc that can be interesting that I forgot to point out is

https://docs.google.com/document/d/1VgnGiVu8vSfm7Et-xAtQYv0PlEpqeyfmhpQUNPmWRJs/edit?usp=sharing
It is the doc I wrote when I submitted Nexmark PR to ease the
reading of the code.
It is not a design doc, I don't know if it belongs to the website
page or to the wiki for beam devs.

Etienne


Le mercredi 06 juin 2018 à 17:48 +0200, Alexey Romanenko a écrit :

FYI: Finally, it was merged and you can find this page here:
https://beam.apache.org/contribute/design-documents/

Thank you everybody who helped me to compile this list!
I’ll try to do my best to update this with new coming docs. In
the same time, please, feel free to add your new docs (or notify
me if I missed this) once they are finished and ready to be
published.

WBR,
Alexey


On 31 May 2018, at 18:52, Eugene Kirpichov mailto:kirpic...@google.com>> wrote:

Thank you!

On Thu, May 31, 2018 at 8:30 AM Alexey Romanenko
mailto:aromanenko@gmail.com>> wrote:

Thank you everybody for provided links. I collected all of them
(please, correct me if I missed something), categorized and
created a dedicated page for Beam website.

Here is a PR for that (please, review):
https://github.com/apache/beam-site/pull/456

WBR,
Alexey


On 30 May 2018, at 13:17, Łukasz Gajowy
mailto:lukasz.gaj...@gmail.com>> wrote:

Hi,

I just wanted to add those two (sorry for being kinda late
with this):


https://docs.google.com/document/d/1dA-5s6OHiP_cz-NRAbwapoKF5MEC1wKps4A5tFbIPKE/edit?usp=sharing

Re: [DISCUSSION] UTests and embedded backends

[Etienne Chauchot]

Hi Robert,
Yes, this is something I really believe in: test coverage offered by embedded 
instances are worth some temporary
flakiness (due to resource over consumption).
I also deeply agree with your point on maintenance: some mocks could hide bugs 
in production code that would cost a lot
in the long term.
Etienne

Le lundi 28 janvier 2019 à 11:44 +0100, Robert Bradshaw a écrit :
> I strongly agree with your original assessment "IMHO I believe thathaving 
> embedded backend for UTests are a lot better
> than mocks." Mocksare sometimes necessary, but in my experience they are 
> often anexpensive (in production and
> maintenance) way to get what amounts tolow true coverage.
> On Mon, Jan 28, 2019 at 11:16 AM Etienne Chauchot  
> wrote:
> 
> Guys,
> I will try using mocks where I see it is needed. As there is a current PR 
> opened on Cassandra, I will take this
> opportunity to add the embedded cassandra server 
> (https://github.com/jsevellec/cassandra-unit) to the UTests.Ticket
> was opened while ago: https://issues.apache.org/jira/browse/BEAM-4164
> Etienne
> Le mardi 22 janvier 2019 à 09:26 +0100, Robert Bradshaw a écrit :
> On Mon, Jan 21, 2019 at 10:42 PM Kenneth Knowles  wrote:
> 
> Robert - you meant this as a mostly-automatic thing that we would engineer, 
> yes?
> 
> Yes, something like TestPipeline that buffers up the pipelines and
> then executes on class teardown (details TBD).
> 
> A lighter-weight fake, like using something in-process sharing a Java 
> interface (versus today a locally running
> service sharing an RPC interface) is still much better than a mock.
> 
> +1
> 
> 
> Kenn
> 
> On Mon, Jan 21, 2019 at 7:17 AM Jean-Baptiste Onofré  
> wrote:
> 
> Hi,
> 
> it makes sense to use embedded backend when:
> 
> 1. it's possible to easily embed the backend
> 2. when the backend is "predictable".
> 
> If it's easy to embed and the backend behavior is predictable, then it
> makes sense.
> In other cases, we can fallback to mock.
> 
> Regards
> JB
> 
> On 21/01/2019 10:07, Etienne Chauchot wrote:
> Hi guys,
> 
> Lately I have been fixing various Elasticsearch flakiness issues in the
> UTests by: introducing timeouts, countdown latches, force refresh,
> embedded cluster size decrease ...
> 
> These flakiness issues are due to the embedded Elasticsearch not coping
> well with the jenkins overload. Still, IMHO I believe that having
> embedded backend for UTests are a lot better than mocks. Even if they
> are less tolerant to load, I prefer having UTests 100% representative of
> real backend and add countermeasures to protect against jenkins overload.
> 
> WDYT ?
> 
> Etienne
> 
> 
> 
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com

Re: Beam Dependency Check Report (2018-06-13)

[Ismaël Mejía]

Hello,

The dependency update report has been working fine. However I found some
issues that I summarized in this issue.
https://issues.apache.org/jira/browse/BEAM-6524
Can Yifan or someone else that knows that area please take a look.

Regards,
Ismaël


On Thu, Jun 14, 2018 at 11:37 PM Yifan Zou  wrote:

> Thank you Paul for letting us know this issue. We will take care of it
> when upgrading dependencies.
>
> On Thu, Jun 14, 2018 at 7:23 AM Paul Gerver  wrote:
>
>> I do have one request to be added to the Java SDK version updates:
>> Beam-3831 [1]. The Google Core depends on the old org.json package which
>> ASF discourages using because of the "Use only for good, not evil" clause.
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-3831
>>
>> On Thu, Jun 14, 2018 at 3:03 AM Etienne Chauchot 
>> wrote:
>>
>>> Thanks Yifan,
>>>
>>> This is great ! It would help us maintain Beam more easily and probably
>>> help us fixing CVE as well.
>>>
>>> Etienne
>>>
>>> Le mercredi 13 juin 2018 à 07:45 -0700, Yifan Zou a écrit :
>>>
>>> Hi,
>>>
>>>
>>> I want to follow up and explain this email.
>>>
>>>
>>> This is a sample email that reports the results of Beam SDK dependency
>>> check, which was proposed here
>>> .
>>> The goal is finding updates for all Beam Python & Java SDKs' dependencies
>>> and prioritize them. The job will be auto triggered in Jenkins once a week
>>> and generate a report. The report lists the high priority updates base on
>>> the following criteria:
>>>
>>>
>>> The dependency update is high priority if:
>>>
>>> 1. It has major versions update available;
>>>
>>>   e.g. org.assertj:assertj-core 2.5.0 -> 3.10.0
>>>
>>>  2. or, it is over 3 minor versions behind the latest version;
>>>
>>>   e.g. org.tukaani:xz 1.5 -> 1.8
>>>
>>> 3. or, the current version is behind the later version for over 180 days.
>>>
>>>
>>>   e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11
>>>
>>>
>>> This job helps Beam contributors to determine the dependency which is
>>> far behind the latest released version. The next step would be automating
>>> filing JIRA bugs for dep updates, group dependencies and identify owners to
>>> take care of the upgrades follow Chamikara's proposal
>>> 
>>> .
>>>
>>>
>>> For more readings:
>>>
>>> [Proposal] Beam dependency check automation
>>> 
>>>  by Yifan Zou
>>>
>>> [Proposal] Beam dependency update policy
>>> 
>>>  by *Chamikara Jayalath*
>>>
>>> Thank you.
>>>
>>> Yifan Zou
>>>
>>> On Wed, Jun 13, 2018 at 7:41 AM Apache Jenkins Server <
>>> jenk...@builds.apache.org> wrote:
>>>
>>> High Priority Dependency Updates Of Beam Python SDK:
>>> *Dependency Name* *Current Version* *Later Version* *Current Version
>>> Release Date* *Later Version Release Date*
>>> google-cloud-bigquery 0.25.0 1.3.0 2017-06-26 2018-06-08
>>> httplib2 0.9.2 0.11.3 2015-09-28 2018-03-30 High Priority Dependency
>>> Updates Of Beam Java SDK:
>>> *Dependency Name* *Current Version* *Later Version* *Current Version
>>> Release Date* *Later Version Release Date*
>>> org.assertj:assertj-core 2.5.0 3.10.0 2016-07-03 2018-05-11
>>> com.google.auto.service:auto-service 1.0-rc2 1.0-rc4 2014-10-24
>>> 2017-12-11
>>> biz.aQute:bndlib 1.43.0 2.0.0.20130123-133441 2011-04-01 2013-02-27
>>> org.apache.cassandra:cassandra-all 3.9 3.11.2 2016-09-26 2018-02-14
>>> commons-cli:commons-cli 1.2 1.4 2009-03-19 2017-03-09
>>> commons-codec:commons-codec 1.9 1.11 2013-12-20 2017-10-17
>>> org.apache.commons:commons-dbcp2 2.1.1 2.3.0 2015-08-02 2018-05-08
>>> com.typesafe:config 1.3.0 1.3.3 2015-05-08 2018-02-21
>>> de.flapdoodle.embed:de.flapdoodle.embed.mongo 1.50.1 2.0.3 2015-12-11
>>> 2018-02-14
>>> de.flapdoodle.embed:de.flapdoodle.embed.process 1.50.1 2.0.3 2015-12-11
>>> 2018-02-14
>>> org.apache.derby:derby 10.12.1.1 10.14.2.0 2015-10-10 2018-05-03
>>> org.apache.derby:derbyclient 10.12.1.1 10.14.2.0 2015-10-10 2018-05-03
>>> org.apache.derby:derbynet 10.12.1.1 10.14.2.0 2015-10-10 2018-05-03
>>> org.elasticsearch:elasticsearch 5.6.3 6.2.4 2017-10-06 2018-04-12
>>> org.elasticsearch:elasticsearch-hadoop 5.0.0 6.2.4 2016-10-26 2018-04-12
>>> org.elasticsearch.client:elasticsearch-rest-client 5.6.3 6.2.4
>>> 2017-10-06 2018-04-12
>>> com.alibaba:fastjson 1.2.12 1.2.47 2016-05-21 2018-03-15
>>> org.elasticsearch.test:framework 5.6.3 6.2.4 2017-10-06 2018-04-12
>>> org.freemarker:freemarker 2.3.25-incubating 2.3.28 2016-06-14 2018-03-30
>>> org.codehaus.groovy:groovy-all 2.4.13 3.0.0-alpha-2 2017-11-22
>>> 2018-04-16
>>> org.apache.hbase:hbase-common 1.2.6 2.0.0.3.0.0.3-2 2017-05-29
>>> 2018-05-31
>>> 

Re: [ANNOUNCE] New committer announcement: Gleb Kanterov

[Maximilian Michels]

Well done Gleb!

On 28.01.19 14:27, Łukasz Gajowy wrote:

Congrats and welcome! :)

pon., 28 sty 2019 o 12:01 Gleb Kanterov > napisał(a):


Thanks to everyone for a warm welcome! Happy to be part of the community.

On Mon, Jan 28, 2019 at 4:43 AM Thomas Weise mailto:t...@apache.org>> wrote:

Congrats!

On Sun, Jan 27, 2019 at 7:19 PM Reza Ardeshir Rokni mailto:raro...@gmail.com>> wrote:

Congratulations!

On Sat, 26 Jan 2019 at 13:50, Pablo Estrada mailto:pabl...@google.com>> wrote:

Congrats:)

On Fri, Jan 25, 2019, 9:24 PM Trần Thành Đạt
mailto:dattran.v...@gmail.com> wrote:

Congratulations Gleb!

On Sat, Jan 26, 2019 at 4:19 AM David Morávek
mailto:david.mora...@gmail.com>>
wrote:

Congratulations!

Sent from my iPhone

On 25 Jan 2019, at 20:41, Kai Jiang mailto:jiang...@gmail.com>> wrote:


Congratulations!

On Fri, Jan 25, 2019 at 10:01 AM Rui Wang
mailto:ruw...@google.com>> wrote:

Congratulations!

-Rui

On Fri, Jan 25, 2019 at 9:58 AM Ruoyun Huang
mailto:ruo...@google.com>> 
wrote:

Congratulations Gleb!

On Fri, Jan 25, 2019 at 9:18 AM Scott Wegner
mailto:sc...@apache.org>>
wrote:

Congrats, and welcome Gleb!

On Fri, Jan 25, 2019 at 9:15 AM Suneel
Marthi mailto:smar...@apache.org>> wrote:

Congratulations

On Fri, Jan 25, 2019 at 12:04 PM Anton
Kedin mailto:ke...@google.com>> wrote:

Congrats!

On Fri, Jan 25, 2019 at 8:54 AM
Ismaël Mejía mailto:ieme...@gmail.com>> wrote:

Well deserved, congratulations
Gleb!

On Fri, Jan 25, 2019 at 10:47
AM Etienne Chauchot
mailto:echauc...@apache.org>>
wrote:
>
> Congrats Gleb and welcome
onboard !
>
> Etienne
>
> Le vendredi 25 janvier 2019
à 10:39 +0100, Alexey
Romanenko a écrit :
>
> Congrats to Gleb and welcome
on board!
>
> On 25 Jan 2019, at 09:22,
Tim Robertson
mailto:timrobertson...@gmail.com>>
wrote:
>
> Welcome Gleb and
congratulations!
>
> On Fri, Jan 25, 2019 at 8:06
AM Kenneth Knowles
mailto:k...@google.com>> wrote:
>
> Hi all,
>
> Please join me and the rest
of the Beam PMC in welcoming a
new committer: Gleb Kanterov
>
> Gleb started contributing to
Beam and quickly dove deep,
doing some sensitive fixes to
 

Re: [ANNOUNCE] New PMC member: Etienne Chauchot

[Łukasz Gajowy]

Thanks for your great work and congratulations! :)

pon., 28 sty 2019 o 12:01 Gleb Kanterov  napisał(a):

> Congratulations Etienne!
>
>
> On Mon, Jan 28, 2019 at 11:36 AM Maximilian Michels 
> wrote:
>
>> Congrats Etienne! It's been great to work with you.
>>
>> On 26.01.19 07:16, Ismaël Mejía wrote:
>> > Congratulations Etienne!
>> >
>> > Le sam. 26 janv. 2019 à 06:42, Reuven Lax > > > a écrit :
>> >
>> > Welcome!
>> >
>> > On Fri, Jan 25, 2019 at 9:30 PM Pablo Estrada > > > wrote:
>> >
>> > Congrats Etienne :)
>> >
>> > On Fri, Jan 25, 2019, 9:24 PM Trần Thành Đạt <
>> dattran.v...@gmail.com
>> >  wrote:
>> >
>> > Congratulations Etienne!
>> >
>> > On Sat, Jan 26, 2019 at 12:08 PM Thomas Weise <
>> t...@apache.org
>> > > wrote:
>> >
>> > Congrats, félicitations!
>> >
>> >
>> > On Fri, Jan 25, 2019 at 3:06 PM Scott Wegner <
>> sc...@apache.org
>> > > wrote:
>> >
>> > Congrats Etienne!
>> >
>> > On Fri, Jan 25, 2019 at 2:34 PM Tim
>> > > > > wrote:
>> >
>> > Congratulations Etienne!
>> >
>> > Tim
>> >
>> >  > On 25 Jan 2019, at 23:00, Kenneth Knowles
>> > mailto:k...@apache.org>>
>> wrote:
>> >  >
>> >  > Hi all,
>> >  >
>> >  > Please join me and the rest of the Beam PMC
>> in
>> > welcoming Etienne Chauchot to join the PMC.
>> >  >
>> >  > Etienne introduced himself to dev@ in
>> September of
>> > 2017 and over the years has contributed to Beam
>> in many
>> > ways - connectors, performance, design
>> discussion,
>> > talks, code reviews, and I'm sure I cannot list
>> them
>> > all. He already has a major impact on the
>> direction of Beam.
>> >  >
>> >  > Thanks for being a part of Beam, Etienne!
>> >  >
>> >  > Kenn
>> >
>> >
>> >
>> > --
>> >
>> >
>> >
>> >
>> > Got feedback? tinyurl.com/swegner-feedback
>> > 
>> >
>>
>
>
> --
> Cheers,
> Gleb
>

Re: [ANNOUNCE] New committer announcement: Gleb Kanterov

[Łukasz Gajowy]

Congrats and welcome! :)

pon., 28 sty 2019 o 12:01 Gleb Kanterov  napisał(a):

> Thanks to everyone for a warm welcome! Happy to be part of the community.
>
> On Mon, Jan 28, 2019 at 4:43 AM Thomas Weise  wrote:
>
>> Congrats!
>>
>> On Sun, Jan 27, 2019 at 7:19 PM Reza Ardeshir Rokni 
>> wrote:
>>
>>> Congratulations!
>>>
>>> On Sat, 26 Jan 2019 at 13:50, Pablo Estrada  wrote:
>>>
 Congrats:)

 On Fri, Jan 25, 2019, 9:24 PM Trần Thành Đạt >>> wrote:

> Congratulations Gleb!
>
> On Sat, Jan 26, 2019 at 4:19 AM David Morávek 
> wrote:
>
>> Congratulations!
>>
>> Sent from my iPhone
>>
>> On 25 Jan 2019, at 20:41, Kai Jiang  wrote:
>>
>> Congratulations!
>>
>> On Fri, Jan 25, 2019 at 10:01 AM Rui Wang  wrote:
>>
>>> Congratulations!
>>>
>>> -Rui
>>>
>>> On Fri, Jan 25, 2019 at 9:58 AM Ruoyun Huang 
>>> wrote:
>>>
 Congratulations Gleb!

 On Fri, Jan 25, 2019 at 9:18 AM Scott Wegner 
 wrote:

> Congrats, and welcome Gleb!
>
> On Fri, Jan 25, 2019 at 9:15 AM Suneel Marthi 
> wrote:
>
>> Congratulations
>>
>> On Fri, Jan 25, 2019 at 12:04 PM Anton Kedin 
>> wrote:
>>
>>> Congrats!
>>>
>>> On Fri, Jan 25, 2019 at 8:54 AM Ismaël Mejía 
>>> wrote:
>>>
 Well deserved, congratulations Gleb!

 On Fri, Jan 25, 2019 at 10:47 AM Etienne Chauchot <
 echauc...@apache.org> wrote:
 >
 > Congrats Gleb and welcome onboard !
 >
 > Etienne
 >
 > Le vendredi 25 janvier 2019 à 10:39 +0100, Alexey Romanenko a
 écrit :
 >
 > Congrats to Gleb and welcome on board!
 >
 > On 25 Jan 2019, at 09:22, Tim Robertson <
 timrobertson...@gmail.com> wrote:
 >
 > Welcome Gleb and congratulations!
 >
 > On Fri, Jan 25, 2019 at 8:06 AM Kenneth Knowles <
 k...@google.com> wrote:
 >
 > Hi all,
 >
 > Please join me and the rest of the Beam PMC in welcoming a
 new committer: Gleb Kanterov
 >
 > Gleb started contributing to Beam and quickly dove deep,
 doing some sensitive fixes to schemas, also general build issues, 
 Beam SQL,
 Avro, and more. In consideration of Gleb's technical and community
 contributions, the Beam PMC trusts Gleb with the responsibilities 
 of a Beam
 committer [1].
 >
 > Thank you, Gleb, for your contributions.
 >
 > Kenn
 >
 > [1]
 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
 >
 >

>>>
>
> --
>
>
>
>
> Got feedback? tinyurl.com/swegner-feedback
>


 --
 
 Ruoyun  Huang


>
> --
> Cheers,
> Gleb
>

Re: BEAM-6324 / #7340: "I've pretty much given up on the PR being merged. I use my own fork for my projects"

[Łukasz Gajowy]

IMHO, I don't think committers spend time watching new PRs coming up, but
they more likely act when pinged. So, we may need some automation in case a
contributor do not use github reviewed proposal. Auto reviewer assignment
seem too much but modifying the PR template to add a sentence such as
"please pickup a reviewer in the proposed list" could be enough.
WDYT ?

and

(1) A sentence in the PR template suggesting adding a reviewer. (easy)


+100! Let's improve the message in the PR template. It costs nothing and
can help a lot. I'd say it should be in bold letters as this is super
important.

Maybe this is also worth reconsidering if auto reviewer assignment (or at
least some form of it) is a bad idea. We can assign committers (the most
"hardcore" option, maybe too much) or ping them in emails/github comments
if there's inactivity in pull requests (the soft one but requires a bot to
be implemented). The way I see this is that such auto assigned reviewer
could always say "I have lots on my plate" but suggest someone else to take
care of the PR. This way the problem that nobody is mentioned by the PR
author is completely gone. Other than that, such an approach feels
efficient to me because it's more "in person" (similar to what Robert
said).

It's certainly disheartening as a
reviewer to put time into reviewing a PR and then the author doesn't
bother to even respond, or (as has happened to me) be told "hey, this
wasn't ready for review yet."

As for "this wasn't ready for review" - there are sometimes situations that
require a PR to be opened before they are actually completed (especially
when working with Jenkins jobs). Given that there might be
misunderstandings authors of such commits should give a clear message
saying "do not merge yet" or "not ready for review" in title or comments or
even close such PR and reopen until the change is ready. It's all about
giving a clear signal to others.

Maybe we should mention it in guidelines/PR message too to avoid situations
like this?

Łukasz


pon., 28 sty 2019 o 11:30 Robert Bradshaw  napisał(a):

> On Mon, Jan 28, 2019 at 10:37 AM Etienne Chauchot 
> wrote:
> >
> > Sure it's a pity than this PR got unnoticed and I think it is a
> combination of factors (PR date around Christmas, the fact that the author
> forgot - AFAIK - to ping a reviewer in either the PR or the ML).
> >
> > I agree with Rui's proposal to enhance visibility of the "how to get a
> reviewed" process.
> >
> > IMHO, I don't think committers spend time watching new PRs coming up,
> but they more likely act when pinged. So, we may need some automation in
> case a contributor do not use github reviewed proposal. Auto reviewer
> assignment seem too much but modifying the PR template to add a sentence
> such as "please pickup a reviewer in the proposed list" could be enough.
> > WDYT ?
>
> +1
>
> I see two somewhat separable areas of improvement:
>
> (1) Getting a reviewer assigned to a PR, and
> (2) Expectations of feedback/timeliness from a reviewer once it has
> been assigned.
>
> The first is the motivation for this thread, but I think we're
> suffering on the second as well.
>
> Given the reactions here, it sounds like most of us are just as
> unhappy this happened as the author, and would be happy to pitch in
> and improve things.
>
> I agree with Kenn that just adding to more the contributor guide
> always help because a contributor guide with everything one might need
> to know is the least likely to actually be read in its entirety.
> Rather it's useful to provide such guidance at the point that it's
> needed. Specifically, I would like to see
>
> (1) A sentence in the PR template suggesting adding a reviewer. (easy)
> (2) An automated recommendation for suggesting good candidate
> reviewers (if we deem Github's suggestions insufficient). (harder)
> (3) A bot that follows up on PRs after 1 week(?) noting the lack of
> action and suggesting (and, implicitly but importantly
> permission/expectation) that the author bring the PR to the list.
> (medium)
>
> We could also have automated emails like the Beam Dependency Check
> Report, but automated emails are much easier to ignore than personal
> ones. Having the author ping dev@ has the added advantage that it
> gives the author something they can do to move the PR forward, and
> provides a clear signal that this is a PR someone cares enough about
> to prioritize it above others. (It's certainly disheartening as a
> reviewer to put time into reviewing a PR and then the author doesn't
> bother to even respond, or (as has happened to me) be told "hey, this
> wasn't ready for review yet." On the other hand it's rewarding to help
> someone, especially a first time contributor, to see their change
> actually get in. Improving this ratio will I think both increase the
> productivity of reviews and the motivation to do them.)
>
> > Also, I started to review the PR on Friday (thx Kenn for pinging me).
> >
> > Etienne
> >
> > Le vendredi 25 janvier 2019 à 

Beam Dependency Check Report (2019-01-28)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
future
0.16.0
0.17.1
2016-10-27
2018-12-10BEAM-5968
oauth2client
3.0.0
4.1.3
2018-12-10
2018-12-10BEAM-6089
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.rabbitmq:amqp-client
4.6.0
5.6.0
2018-03-26
2019-01-25BEAM-5895
org.apache.rat:apache-rat-tasks
0.12
0.13
2016-06-07
2018-10-13BEAM-6039
com.google.auto.service:auto-service
1.0-rc2
1.0-rc4
2014-10-25
2017-12-11BEAM-5541
com.amazonaws:aws-java-sdk-kinesis
1.11.255
1.11.490
2017-12-23
2019-01-26BEAM-6330
com.gradle:build-scan-plugin
1.13.1
2.1
2018-04-10
2018-12-10BEAM-5543
org.conscrypt:conscrypt-openjdk
1.1.3
1.4.2
2018-06-04
2018-12-11BEAM-5748
org.elasticsearch:elasticsearch
6.4.0
7.0.0-alpha2
2018-08-18
2018-12-17BEAM-6090
org.elasticsearch:elasticsearch-hadoop
5.0.0
7.0.0-alpha2
2016-10-26
2018-12-17BEAM-5551
org.elasticsearch.client:elasticsearch-rest-client
6.4.0
7.0.0-alpha2
2018-08-18
2018-12-17BEAM-6091
org.elasticsearch.test:framework
6.4.0
7.0.0-alpha2
2018-08-18
2018-12-17BEAM-6092
com.google.auth:google-auth-library-credentials
0.10.0
0.13.0
2018-07-12
2019-01-17BEAM-6478
gradle.plugin.io.pry.gradle.offline_dependencies:gradle-offline-dependencies-plugin
0.3
0.5.0
2018-01-05
2019-01-07BEAM-6377
net.researchgate:gradle-release
2.6.0
2.8.0
2017-05-22
2018-12-31BEAM-6331
io.grpc:grpc-auth
1.13.1
1.18.0
2018-06-21
2019-01-15BEAM-5896
io.grpc:grpc-context
1.13.1
1.18.0
2018-06-21
2019-01-15BEAM-5897
io.grpc:grpc-core
1.13.1
1.18.0
2018-06-21
2019-01-15BEAM-5898
io.grpc:grpc-netty
1.13.1
1.18.0
2018-06-21
2019-01-15BEAM-5899
io.grpc:grpc-protobuf
1.13.1
1.18.0
2018-06-21
2019-01-15BEAM-5900
io.grpc:grpc-stub
1.13.1
1.18.0
2018-06-21
2019-01-15BEAM-5901
io.grpc:grpc-testing
1.13.1
1.18.0
2018-06-21
2019-01-15BEAM-5902
com.google.code.gson:gson
2.7
2.8.5
2016-06-14
2018-05-22BEAM-5558
org.apache.hbase:hbase-common
1.2.6
2.1.2
2017-05-29
2018-12-29BEAM-5560
org.apache.hbase:hbase-hadoop-compat
1.2.6
2.1.2
2017-05-29
2018-12-29BEAM-5561
org.apache.hbase:hbase-hadoop2-compat
1.2.6
2.1.2
2017-05-29
2018-12-29BEAM-5562
org.apache.hbase:hbase-server
1.2.6
2.1.2
2017-05-29
2018-12-29BEAM-5563
org.apache.hbase:hbase-shaded-client
1.2.6
2.1.2
2017-05-29
2018-12-29BEAM-5564
org.apache.hive:hive-cli
2.1.0
3.1.1
2016-06-17
2018-10-24BEAM-5566
org.apache.hive:hive-common
2.1.0
3.1.1
2016-06-17
2018-10-24BEAM-5567
org.apache.hive:hive-exec
2.1.0
3.1.1
2016-06-17
2018-10-24BEAM-5568
org.apache.hive.hcatalog:hive-hcatalog-core
2.1.0
3.1.1
2016-06-17
2018-10-24BEAM-5569
net.java.dev.javacc:javacc
4.0
7.0.4
2006-03-17
2018-09-17BEAM-5570
javax.servlet:javax.servlet-api
3.1.0
4.0.1
2013-04-25
2018-04-20BEAM-5750
redis.clients:jedis
2.9.0
3.0.1
2016-07-22
2018-12-27BEAM-6125
org.eclipse.jetty:jetty-server
9.2.10.v20150310
9.4.14.v20181114
2015-03-10
2018-11-14BEAM-5752
org.eclipse.jetty:jetty-servlet
9.2.10.v20150310
9.4.14.v20181114
2015-03-10
2018-11-14BEAM-5753
net.java.dev.jna:jna
4.1.0
5.2.0
2014-03-06
2018-12-23BEAM-5573
com.esotericsoftware:kryo
4.0.2
5.0.0-RC1
2018-03-20
2018-06-19BEAM-5809

BigQuery TimePartitioning IT

[Wout Scheepers]

Hey all,

For my BigQuery clustering PR I wrote an integration test to test time 
partitioning and clustering [1].
Can anyone create the “BigQueryTimePartitioningIT” dataset in the beam testing 
project (apache-beam-testing)? Or is dataset creation included somehow in the 
gradle setup?

Thanks,
Wout

[1] 
https://github.com/apache/beam/pull/7061/commits/2df3611a0c1afa4602b21dd655e85fae4f30200d#diff-00c26efe2f7d595ee31affd11a8cbce2

Re: [ANNOUNCE] New PMC member: Etienne Chauchot

[Gleb Kanterov]

Congratulations Etienne!


On Mon, Jan 28, 2019 at 11:36 AM Maximilian Michels  wrote:

> Congrats Etienne! It's been great to work with you.
>
> On 26.01.19 07:16, Ismaël Mejía wrote:
> > Congratulations Etienne!
> >
> > Le sam. 26 janv. 2019 à 06:42, Reuven Lax  > > a écrit :
> >
> > Welcome!
> >
> > On Fri, Jan 25, 2019 at 9:30 PM Pablo Estrada  > > wrote:
> >
> > Congrats Etienne :)
> >
> > On Fri, Jan 25, 2019, 9:24 PM Trần Thành Đạt <
> dattran.v...@gmail.com
> >  wrote:
> >
> > Congratulations Etienne!
> >
> > On Sat, Jan 26, 2019 at 12:08 PM Thomas Weise <
> t...@apache.org
> > > wrote:
> >
> > Congrats, félicitations!
> >
> >
> > On Fri, Jan 25, 2019 at 3:06 PM Scott Wegner <
> sc...@apache.org
> > > wrote:
> >
> > Congrats Etienne!
> >
> > On Fri, Jan 25, 2019 at 2:34 PM Tim
> >  > > wrote:
> >
> > Congratulations Etienne!
> >
> > Tim
> >
> >  > On 25 Jan 2019, at 23:00, Kenneth Knowles
> > mailto:k...@apache.org>>
> wrote:
> >  >
> >  > Hi all,
> >  >
> >  > Please join me and the rest of the Beam PMC in
> > welcoming Etienne Chauchot to join the PMC.
> >  >
> >  > Etienne introduced himself to dev@ in
> September of
> > 2017 and over the years has contributed to Beam
> in many
> > ways - connectors, performance, design
> discussion,
> > talks, code reviews, and I'm sure I cannot list
> them
> > all. He already has a major impact on the
> direction of Beam.
> >  >
> >  > Thanks for being a part of Beam, Etienne!
> >  >
> >  > Kenn
> >
> >
> >
> > --
> >
> >
> >
> >
> > Got feedback? tinyurl.com/swegner-feedback
> > 
> >
>


-- 
Cheers,
Gleb

Re: [ANNOUNCE] New committer announcement: Gleb Kanterov

[Gleb Kanterov]

Thanks to everyone for a warm welcome! Happy to be part of the community.

On Mon, Jan 28, 2019 at 4:43 AM Thomas Weise  wrote:

> Congrats!
>
> On Sun, Jan 27, 2019 at 7:19 PM Reza Ardeshir Rokni 
> wrote:
>
>> Congratulations!
>>
>> On Sat, 26 Jan 2019 at 13:50, Pablo Estrada  wrote:
>>
>>> Congrats:)
>>>
>>> On Fri, Jan 25, 2019, 9:24 PM Trần Thành Đạt >> wrote:
>>>
 Congratulations Gleb!

 On Sat, Jan 26, 2019 at 4:19 AM David Morávek 
 wrote:

> Congratulations!
>
> Sent from my iPhone
>
> On 25 Jan 2019, at 20:41, Kai Jiang  wrote:
>
> Congratulations!
>
> On Fri, Jan 25, 2019 at 10:01 AM Rui Wang  wrote:
>
>> Congratulations!
>>
>> -Rui
>>
>> On Fri, Jan 25, 2019 at 9:58 AM Ruoyun Huang 
>> wrote:
>>
>>> Congratulations Gleb!
>>>
>>> On Fri, Jan 25, 2019 at 9:18 AM Scott Wegner 
>>> wrote:
>>>
 Congrats, and welcome Gleb!

 On Fri, Jan 25, 2019 at 9:15 AM Suneel Marthi 
 wrote:

> Congratulations
>
> On Fri, Jan 25, 2019 at 12:04 PM Anton Kedin 
> wrote:
>
>> Congrats!
>>
>> On Fri, Jan 25, 2019 at 8:54 AM Ismaël Mejía 
>> wrote:
>>
>>> Well deserved, congratulations Gleb!
>>>
>>> On Fri, Jan 25, 2019 at 10:47 AM Etienne Chauchot <
>>> echauc...@apache.org> wrote:
>>> >
>>> > Congrats Gleb and welcome onboard !
>>> >
>>> > Etienne
>>> >
>>> > Le vendredi 25 janvier 2019 à 10:39 +0100, Alexey Romanenko a
>>> écrit :
>>> >
>>> > Congrats to Gleb and welcome on board!
>>> >
>>> > On 25 Jan 2019, at 09:22, Tim Robertson <
>>> timrobertson...@gmail.com> wrote:
>>> >
>>> > Welcome Gleb and congratulations!
>>> >
>>> > On Fri, Jan 25, 2019 at 8:06 AM Kenneth Knowles <
>>> k...@google.com> wrote:
>>> >
>>> > Hi all,
>>> >
>>> > Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Gleb Kanterov
>>> >
>>> > Gleb started contributing to Beam and quickly dove deep, doing
>>> some sensitive fixes to schemas, also general build issues, Beam 
>>> SQL, Avro,
>>> and more. In consideration of Gleb's technical and community 
>>> contributions,
>>> the Beam PMC trusts Gleb with the responsibilities of a Beam 
>>> committer [1].
>>> >
>>> > Thank you, Gleb, for your contributions.
>>> >
>>> > Kenn
>>> >
>>> > [1]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>> >
>>> >
>>>
>>

 --




 Got feedback? tinyurl.com/swegner-feedback

>>>
>>>
>>> --
>>> 
>>> Ruoyun  Huang
>>>
>>>

-- 
Cheers,
Gleb

Re: [DISCUSSION] UTests and embedded backends

[Robert Bradshaw]

I strongly agree with your original assessment "IMHO I believe that
having embedded backend for UTests are a lot better than mocks." Mocks
are sometimes necessary, but in my experience they are often an
expensive (in production and maintenance) way to get what amounts to
low true coverage.

On Mon, Jan 28, 2019 at 11:16 AM Etienne Chauchot  wrote:
>
> Guys,
>
> I will try using mocks where I see it is needed. As there is a current PR 
> opened on Cassandra, I will take this opportunity to add the embedded 
> cassandra server (https://github.com/jsevellec/cassandra-unit) to the UTests.
> Ticket was opened while ago: https://issues.apache.org/jira/browse/BEAM-4164
>
> Etienne
>
> Le mardi 22 janvier 2019 à 09:26 +0100, Robert Bradshaw a écrit :
>
> On Mon, Jan 21, 2019 at 10:42 PM Kenneth Knowles  wrote:
>
>
> Robert - you meant this as a mostly-automatic thing that we would engineer, 
> yes?
>
>
> Yes, something like TestPipeline that buffers up the pipelines and
>
> then executes on class teardown (details TBD).
>
>
> A lighter-weight fake, like using something in-process sharing a Java 
> interface (versus today a locally running service sharing an RPC interface) 
> is still much better than a mock.
>
>
> +1
>
>
>
> Kenn
>
>
> On Mon, Jan 21, 2019 at 7:17 AM Jean-Baptiste Onofré  
> wrote:
>
>
> Hi,
>
>
> it makes sense to use embedded backend when:
>
>
> 1. it's possible to easily embed the backend
>
> 2. when the backend is "predictable".
>
>
> If it's easy to embed and the backend behavior is predictable, then it
>
> makes sense.
>
> In other cases, we can fallback to mock.
>
>
> Regards
>
> JB
>
>
> On 21/01/2019 10:07, Etienne Chauchot wrote:
>
> Hi guys,
>
>
> Lately I have been fixing various Elasticsearch flakiness issues in the
>
> UTests by: introducing timeouts, countdown latches, force refresh,
>
> embedded cluster size decrease ...
>
>
> These flakiness issues are due to the embedded Elasticsearch not coping
>
> well with the jenkins overload. Still, IMHO I believe that having
>
> embedded backend for UTests are a lot better than mocks. Even if they
>
> are less tolerant to load, I prefer having UTests 100% representative of
>
> real backend and add countermeasures to protect against jenkins overload.
>
>
> WDYT ?
>
>
> Etienne
>
>
>
>
> --
>
> Jean-Baptiste Onofré
>
> jbono...@apache.org
>
> http://blog.nanthrax.net
>
> Talend - http://www.talend.com

Re: [ANNOUNCE] New PMC member: Etienne Chauchot

[Maximilian Michels]

Congrats Etienne! It's been great to work with you.

On 26.01.19 07:16, Ismaël Mejía wrote:

Congratulations Etienne!

Le sam. 26 janv. 2019 à 06:42, Reuven Lax > a écrit :


Welcome!

On Fri, Jan 25, 2019 at 9:30 PM Pablo Estrada mailto:pabl...@google.com>> wrote:

Congrats Etienne :)

On Fri, Jan 25, 2019, 9:24 PM Trần Thành Đạt mailto:dattran.v...@gmail.com> wrote:

Congratulations Etienne!

On Sat, Jan 26, 2019 at 12:08 PM Thomas Weise mailto:t...@apache.org>> wrote:

Congrats, félicitations!


On Fri, Jan 25, 2019 at 3:06 PM Scott Wegner mailto:sc...@apache.org>> wrote:

Congrats Etienne!

On Fri, Jan 25, 2019 at 2:34 PM Tim
mailto:timrobertson...@gmail.com>> wrote:

Congratulations Etienne!

Tim

 > On 25 Jan 2019, at 23:00, Kenneth Knowles
mailto:k...@apache.org>> wrote:
 >
 > Hi all,
 >
 > Please join me and the rest of the Beam PMC in
welcoming Etienne Chauchot to join the PMC.
 >
 > Etienne introduced himself to dev@ in September of
2017 and over the years has contributed to Beam in many
ways - connectors, performance, design discussion,
talks, code reviews, and I'm sure I cannot list them
all. He already has a major impact on the direction of 
Beam.
 >
 > Thanks for being a part of Beam, Etienne!
 >
 > Kenn



-- 





Got feedback? tinyurl.com/swegner-feedback

Re: [ANNOUNCE] New PMC member: Etienne Chauchot

[Robert Bradshaw]

Thanks for all your great work. Congratulations and welcome!

On Mon, Jan 28, 2019 at 10:21 AM Alexey Romanenko
 wrote:
>
> Great job! Congrats, Etienne!
>
> On 28 Jan 2019, at 07:18, Ahmet Altay  wrote:
>
> Congratulations Etienne!
>
> On Sun, Jan 27, 2019 at 7:15 PM Reza Ardeshir Rokni  wrote:
>>
>> Congratulations Etienne!
>>
>> On Sat, 26 Jan 2019 at 14:16, Ismaël Mejía  wrote:
>>>
>>> Congratulations Etienne!
>>>
>>> Le sam. 26 janv. 2019 à 06:42, Reuven Lax  a écrit :

 Welcome!

 On Fri, Jan 25, 2019 at 9:30 PM Pablo Estrada  wrote:
>
> Congrats Etienne :)
>
> On Fri, Jan 25, 2019, 9:24 PM Trần Thành Đạt  wrote:
>>
>> Congratulations Etienne!
>>
>> On Sat, Jan 26, 2019 at 12:08 PM Thomas Weise  wrote:
>>>
>>> Congrats, félicitations!
>>>
>>>
>>> On Fri, Jan 25, 2019 at 3:06 PM Scott Wegner  wrote:

 Congrats Etienne!

 On Fri, Jan 25, 2019 at 2:34 PM Tim  wrote:
>
> Congratulations Etienne!
>
> Tim
>
> > On 25 Jan 2019, at 23:00, Kenneth Knowles  wrote:
> >
> > Hi all,
> >
> > Please join me and the rest of the Beam PMC in welcoming Etienne 
> > Chauchot to join the PMC.
> >
> > Etienne introduced himself to dev@ in September of 2017 and over 
> > the years has contributed to Beam in many ways - connectors, 
> > performance, design discussion, talks, code reviews, and I'm sure I 
> > cannot list them all. He already has a major impact on the 
> > direction of Beam.
> >
> > Thanks for being a part of Beam, Etienne!
> >
> > Kenn



 --




 Got feedback? tinyurl.com/swegner-feedback
>
>

Re: BEAM-6324 / #7340: "I've pretty much given up on the PR being merged. I use my own fork for my projects"

[Robert Bradshaw]

On Mon, Jan 28, 2019 at 10:37 AM Etienne Chauchot  wrote:
>
> Sure it's a pity than this PR got unnoticed and I think it is a combination 
> of factors (PR date around Christmas, the fact that the author forgot - AFAIK 
> - to ping a reviewer in either the PR or the ML).
>
> I agree with Rui's proposal to enhance visibility of the "how to get a 
> reviewed" process.
>
> IMHO, I don't think committers spend time watching new PRs coming up, but 
> they more likely act when pinged. So, we may need some automation in case a 
> contributor do not use github reviewed proposal. Auto reviewer assignment 
> seem too much but modifying the PR template to add a sentence such as "please 
> pickup a reviewer in the proposed list" could be enough.
> WDYT ?

+1

I see two somewhat separable areas of improvement:

(1) Getting a reviewer assigned to a PR, and
(2) Expectations of feedback/timeliness from a reviewer once it has
been assigned.

The first is the motivation for this thread, but I think we're
suffering on the second as well.

Given the reactions here, it sounds like most of us are just as
unhappy this happened as the author, and would be happy to pitch in
and improve things.

I agree with Kenn that just adding to more the contributor guide
always help because a contributor guide with everything one might need
to know is the least likely to actually be read in its entirety.
Rather it's useful to provide such guidance at the point that it's
needed. Specifically, I would like to see

(1) A sentence in the PR template suggesting adding a reviewer. (easy)
(2) An automated recommendation for suggesting good candidate
reviewers (if we deem Github's suggestions insufficient). (harder)
(3) A bot that follows up on PRs after 1 week(?) noting the lack of
action and suggesting (and, implicitly but importantly
permission/expectation) that the author bring the PR to the list.
(medium)

We could also have automated emails like the Beam Dependency Check
Report, but automated emails are much easier to ignore than personal
ones. Having the author ping dev@ has the added advantage that it
gives the author something they can do to move the PR forward, and
provides a clear signal that this is a PR someone cares enough about
to prioritize it above others. (It's certainly disheartening as a
reviewer to put time into reviewing a PR and then the author doesn't
bother to even respond, or (as has happened to me) be told "hey, this
wasn't ready for review yet." On the other hand it's rewarding to help
someone, especially a first time contributor, to see their change
actually get in. Improving this ratio will I think both increase the
productivity of reviews and the motivation to do them.)

> Also, I started to review the PR on Friday (thx Kenn for pinging me).
>
> Etienne
>
> Le vendredi 25 janvier 2019 à 10:21 -0800, Rui Wang a écrit :
>
> We have code contribution guidelines [1] and it says useful tips to make PR 
> reviewed and merged. But I guess it hides in Beam website so new contributors 
> are likely to ignore it. In order to make the guidance easy to find and read 
> for new contributors, we probably can
>
> a. Move number 5 item from [1] to a separate section and name it "Tips to get 
> your PR reviewed and merged"
> b. Put the link to the Github pull request template, so when a contributor 
> creates the first PR, the contributor could see the link (or even paste text 
> from contribution guide). It will be a good chance that new contributors read 
> what's in pull request template.
>
>
> -Rui
>
> [1] https://beam.apache.org/contribute/#make-your-change
>
> On Fri, Jan 25, 2019 at 9:24 AM Alexey Romanenko  
> wrote:
>
> For sure, it’s a pity that this PR has not been addressed for a long time (I 
> guess, we probably have other ones like this) but, as I can see from this PR 
> history, review has not been requested explicitly by author (and this is one 
> of the our recommendations for code contribution [1]).
>
> What are the options to improve this:
>
> 1) Make it more clearly for new contributors that they need to ask for a 
> review explicitly (with a help of recommendations that already provided in 
> top-right corner on PR page)
> 2) Create a bot (like “stale” bot that we have) to check for non-addressed 
> PRs that are more than, say, 7 days, and send notification to dev@ (or 
> dedicated, see n.3) mailing list if they are starving for review.
> 3) (Optionally) Create new mailing list called pr@ for new coming and 
> non-addressed PRs
>
> [1] https://beam.apache.org/contribute/#make-your-change
>
>
> On 25 Jan 2019, at 17:50, Ismaël Mejía  wrote:
>
> The fact that this happened is a real pity. However it is clearly an
> exception and not the rule. Really few PRs have been long time without
> review. Can we somehow automatically send a notification if a PR has
> no assigned reviewers, or if it has not been reviewed after some time
> as Tim suggested?
>
> On Fri, Jan 25, 2019 at 9:43 AM Tim Robertson  
> wrote:
>
>
> Thanks 

Re: [DISCUSSION] UTests and embedded backends

[Etienne Chauchot]

Guys,
I will try using mocks where I see it is needed. As there is a current PR 
opened on Cassandra, I will take this
opportunity to add the embedded cassandra server 
(https://github.com/jsevellec/cassandra-unit) to the UTests.Ticket was
opened while ago: https://issues.apache.org/jira/browse/BEAM-4164 
id="-x-evo-selection-start-marker">
Etienne
Le mardi 22 janvier 2019 à 09:26 +0100, Robert Bradshaw a écrit :
> On Mon, Jan 21, 2019 at 10:42 PM Kenneth Knowles  wrote:
> 
> Robert - you meant this as a mostly-automatic thing that we would engineer, 
> yes?
> Yes, something like TestPipeline that buffers up the pipelines andthen 
> executes on class teardown (details TBD).
> A lighter-weight fake, like using something in-process sharing a Java 
> interface (versus today a locally running
> service sharing an RPC interface) is still much better than a mock.
> +1
> 
> Kenn
> On Mon, Jan 21, 2019 at 7:17 AM Jean-Baptiste Onofré  
> wrote:
> 
> Hi,
> it makes sense to use embedded backend when:
> 1. it's possible to easily embed the backend2. when the backend is 
> "predictable".
> If it's easy to embed and the backend behavior is predictable, then itmakes 
> sense.In other cases, we can fallback to
> mock.
> RegardsJB
> On 21/01/2019 10:07, Etienne Chauchot wrote:
> Hi guys,
> Lately I have been fixing various Elasticsearch flakiness issues in theUTests 
> by: introducing timeouts, countdown
> latches, force refresh,embedded cluster size decrease ...
> These flakiness issues are due to the embedded Elasticsearch not copingwell 
> with the jenkins overload. Still, IMHO I
> believe that havingembedded backend for UTests are a lot better than mocks. 
> Even if theyare less tolerant to load, I
> prefer having UTests 100% representative ofreal backend and add 
> countermeasures to protect against jenkins overload.
> WDYT ?
> Etienne
> 
> 
> --Jean-Baptiste Onofréjbonofre@apache.orghttp://blog.nanthrax.netTalend - 
> http://www.talend.com

Re: BEAM-6324 / #7340: "I've pretty much given up on the PR being merged. I use my own fork for my projects"

[Etienne Chauchot]

Sure it's a pity than this PR got unnoticed and I think it is a combination of 
factors (PR date around Christmas, the
fact that the author forgot - AFAIK - to ping a reviewer in either the PR or 
the ML).
I agree with Rui's proposal to enhance visibility of the "how to get a 
reviewed" process.
IMHO, I don't think committers spend time watching new PRs coming up, but they 
more likely act when pinged. So, we may
need some automation in case a contributor do not use github reviewed proposal. 
Auto reviewer assignment seem too much
but modifying the PR template to add a sentence such as "please pickup a 
reviewer in the proposed list" could be
enough. WDYT ? 

Also, I started to review the PR on Friday (thx Kenn for pinging me).
Etienne
Le vendredi 25 janvier 2019 à 10:21 -0800, Rui Wang a écrit :
> We have code contribution guidelines [1] and it says useful tips to make PR 
> reviewed and merged. But I guess it hides
> in Beam website so new contributors are likely to ignore it. In order to make 
> the guidance easy to find and read for
> new contributors, we probably can
> 
> a. Move number 5 item from [1] to a separate section and name it "Tips to get 
> your PR reviewed and merged"
> b. Put the link to the Github pull request template, so when a contributor 
> creates the first PR, the contributor could
> see the link (or even paste text from contribution guide). It will be a good 
> chance that new contributors read what's
> in pull request template.
> 
> 
> -Rui
> 
> [1] https://beam.apache.org/contribute/#make-your-change
> On Fri, Jan 25, 2019 at 9:24 AM Alexey Romanenko  
> wrote:
> > For sure, it’s a pity that this PR has not been addressed for a long time 
> > (I guess, we probably have other ones like
> > this) but, as I can see from this PR history, review has not been requested 
> > explicitly by author (and this is one of
> > the our recommendations for code contribution [1]).
> > What are the options to improve this:
> > 
> > 1) Make it more clearly for new contributors that they need to ask for a 
> > review explicitly (with a help of
> > recommendations that already provided in top-right corner on PR page)
> > 2) Create a bot (like “stale” bot that we have) to check for non-addressed 
> > PRs that are more than, say, 7 days, and
> > send notification to dev@ (or dedicated, see n.3) mailing list if they are 
> > starving for review.
> > 3) (Optionally) Create new mailing list called pr@ for new coming and 
> > non-addressed PRs
> > 
> > [1] https://beam.apache.org/contribute/#make-your-change
> > 
> > 
> > > On 25 Jan 2019, at 17:50, Ismaël Mejía  wrote:
> > > 
> > > The fact that this happened is a real pity. However it is clearly an
> > > exception and not the rule. Really few PRs have been long time without
> > > review. Can we somehow automatically send a notification if a PR has
> > > no assigned reviewers, or if it has not been reviewed after some time
> > > as Tim suggested?
> > > 
> > > On Fri, Jan 25, 2019 at 9:43 AM Tim Robertson  
> > > wrote:
> > > > Thanks Kenn
> > > > 
> > > > I tend to think that timing is the main contributing factor as you note 
> > > > on the Jira - it slipped down with no
> > > > reminders / bumps sent on any channels that I can see.
> > > > 
> > > > Would something that alerts the dev@ list of PRs that have not received 
> > > > any attention after N days be helpful
> > > > perhaps?
> > > > Even if that only prompts action by one of us to comment on the PR that 
> > > > it's been acknowledged would likely be
> > > > enough to engage the contributor - they would hopefully then ping the 
> > > > individual if it then slips for a long
> > > > time.
> > > > 
> > > > Next week will be my first I'll be able to work on Beam in 2019, but 
> > > > I'll comment on that PR now too as it's
> > > > missing tests.
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > On Fri, Jan 25, 2019 at 7:27 AM Kenneth Knowles  wrote:
> > > > > The subject line is a quote from BEAM-6324*
> > > > > 
> > > > > This makes me sad. I hope/expect it is a failure to route a pull 
> > > > > request to the right reviewer. I am less sad
> > > > > about the functionality than the sentiment and how a contributor is 
> > > > > being discouraged.
> > > > > 
> > > > > Does anyone have ideas that could help?
> > > > > 
> > > > > Kenn
> > > > > 
> > > > > *https://issues.apache.org/jira/browse/BEAM-6324

Re: [ANNOUNCE] New PMC member: Etienne Chauchot

[Alexey Romanenko]

Great job! Congrats, Etienne! 

> On 28 Jan 2019, at 07:18, Ahmet Altay  wrote:
> 
> Congratulations Etienne!
> 
> On Sun, Jan 27, 2019 at 7:15 PM Reza Ardeshir Rokni  > wrote:
> Congratulations Etienne! 
> 
> On Sat, 26 Jan 2019 at 14:16, Ismaël Mejía  > wrote:
> Congratulations Etienne!
> 
> Le sam. 26 janv. 2019 à 06:42, Reuven Lax  > a écrit :
> Welcome!
> 
> On Fri, Jan 25, 2019 at 9:30 PM Pablo Estrada  > wrote:
> Congrats Etienne :)
> 
> On Fri, Jan 25, 2019, 9:24 PM Trần Thành Đạt   wrote:
> Congratulations Etienne!  
> 
> On Sat, Jan 26, 2019 at 12:08 PM Thomas Weise  > wrote:
> Congrats, félicitations!
> 
> 
> On Fri, Jan 25, 2019 at 3:06 PM Scott Wegner  > wrote:
> Congrats Etienne!
> 
> On Fri, Jan 25, 2019 at 2:34 PM Tim  > wrote:
> Congratulations Etienne!
> 
> Tim
> 
> > On 25 Jan 2019, at 23:00, Kenneth Knowles  > > wrote:
> > 
> > Hi all,
> > 
> > Please join me and the rest of the Beam PMC in welcoming Etienne Chauchot 
> > to join the PMC.
> > 
> > Etienne introduced himself to dev@ in September of 2017 and over the years 
> > has contributed to Beam in many ways - connectors, performance, design 
> > discussion, talks, code reviews, and I'm sure I cannot list them all. He 
> > already has a major impact on the direction of Beam.
> > 
> > Thanks for being a part of Beam, Etienne!
> > 
> > Kenn
> 
> 
> -- 
> 
> 
> 
> 
> Got feedback? tinyurl.com/swegner-feedback 
>