Re: Apache Beam Newsletter - February/March 2019

[Kenneth Knowles]

On Tue, Mar 12, 2019 at 9:04 PM Thomas Weise  wrote:

> The release blog is already on the radar for improvement [1].
>
> I don't think there is a need to separate out the release blogs. That's if
> they provide content that is valuable to users (IMO that's not exactly the
> case today).
>

The case I am thinking about is a user searching the web for an issue and
figuring out what version it was fixed in. Dep upgrades being a primary
thing to notice.


> If release blogs are just reformatted JIRA reports, then maybe we can skip
> them altogether (release notes are already linked from download page).
>

One reason is that Jira remains mutable and I am not so sure of the SEO, in
terms of helping users figure out if an upgrade might solve their problem.


> In that case I would much rather like to see the original JIRA report
> cleaned up as part of the release process.
>

Either way, this seems worthwhile. I think release managers often do this
anyhow. How could we formalize it? Do we need to? My favorite form of
formal process is just to get more eyes on some LGTM.


> On the other hand, if release blogs are assembled similar to the
> newsletter that we discuss here, through broader contributor input and with
> the aim to provide more meaningful content to users, then I don't see why
> we need to put them into a different area.
>

Same answer: people searching for issue resolution. I wonder if our release
notes should be a static copy/paste of the Jira list, deleting things that
really make no sense and editing everything else to be meaningful. But I
get the idea that we don't really need a separate area.

What if newsletters matched releases, and could be drafted throughout the 6
week period, with folks really trying to give a narrative to what they
contributed to a release?

Kenn


Overall, given proposed newsletter cadence of 2 months and existing release
> cadence, we would probably still end up with a monthly piece of news.
>
> Thanks,
> Thomas
>
>
>
> https://lists.apache.org/thread.html/224908775d8807eecc53e45f14f99d6a439beb8bb5165e3a813bf82d@%3Cdev.beam.apache.org%3E
>
>
> On Tue, Mar 12, 2019 at 9:27 AM Rose Nguyen  wrote:
>
>> Once every 2 months works. We include both big and small items. It'll
>> provide the focus Thomas mentions but still catches the frequency that
>> Pablo suggested for relevancy. To Alexey's point, it's difficult to
>> navigate the more recent non-release blog posts (<1 year) because they are
>> sprinkled in.
>>
>> A compromise for all of our points is to move the release notes to a
>> separate section, and have a single section that's blog/news.
>>
>> And thanks for wanting to contribute, Aizhamal! Teaming up with you will
>> make things run smoothly.
>>
>> On Tue, Mar 12, 2019 at 3:58 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> +1 for “News” section. Though, I’d propose to exclude “release notes”
>>> posts from “Blog” section list (since now we have quite regular releases,
>>> it makes blog posts list not very readable for users) and move them to new
>>> created News section. Also, this section could include the posts about
>>> other Beam events, like meet-ups, conferences, project updates, etc. I’d
>>> keep “Blog” section for more technical posts.
>>>
>>> On 12 Mar 2019, at 06:31, Pablo Estrada  wrote:
>>>
>>> I agree that the newsletter fits well as a blog post. I think that'd
>>> work best.
>>>
>>> As for the cadence, I think quarterly would be a bit too infrequent. I
>>> like once a month, or once every other month to have at least one per
>>> release. Though happy to hear other people's thoughts.
>>> Best
>>> -P.
>>>
>>> On Mon, Mar 11, 2019 at 6:42 PM Thomas Weise  wrote:
>>>
 +1 for single blog/news section

 Also I wouldn't mind quarterly cadence, to provide more focus for folks
 to contribute.

 On Mon, Mar 11, 2019 at 6:35 PM Kenneth Knowles 
 wrote:

> Could the newsletter be a blog entry? If you check out
> https://blogs.apache.org/ many of the posts are "Apache News
> Round-up". We could rename the "Blog" section to "News" if you ask me.
>
> Kenn
>
> On Mon, Mar 11, 2019 at 5:13 PM Aizhamal Nurmamat kyzy <
> aizha...@google.com> wrote:
>
>> Hello everyone,
>>
>> I had a chat with Rose on how I can support the effort and keep
>> sending the newsletters on a monthly basis. The new workflow would look 
>> as
>> follows:
>>
>>1. Send out the same [CALL FOR ITEMS] where you can contribute to
>>the Google doc with deadlines
>>2. Edit the doc after the deadline
>>3. Convert the file into Markdown
>>4. Send in PR to add the file to Beam repo in Newsletter directory
>>5. Have people send their fixes/updates through PRs
>>
>> In this effort, I can support Rose in steps 3 & 4.
>>
>> We would also need:
>>
>>- Create a Newsletter section under the Community tab
>>- Write guid

Re: Apache Beam Newsletter - February/March 2019

[Thomas Weise]

The release blog is already on the radar for improvement [1].

I don't think there is a need to separate out the release blogs. That's if
they provide content that is valuable to users (IMO that's not exactly the
case today).

If release blogs are just reformatted JIRA reports, then maybe we can skip
them altogether (release notes are already linked from download page). In
that case I would much rather like to see the original JIRA report cleaned
up as part of the release process.

On the other hand, if release blogs are assembled similar to the newsletter
that we discuss here, through broader contributor input and with the aim to
provide more meaningful content to users, then I don't see why we need to
put them into a different area.

Overall, given proposed newsletter cadence of 2 months and existing release
cadence, we would probably still end up with a monthly piece of news.

Thanks,
Thomas


https://lists.apache.org/thread.html/224908775d8807eecc53e45f14f99d6a439beb8bb5165e3a813bf82d@%3Cdev.beam.apache.org%3E


On Tue, Mar 12, 2019 at 9:27 AM Rose Nguyen  wrote:

> Once every 2 months works. We include both big and small items. It'll
> provide the focus Thomas mentions but still catches the frequency that
> Pablo suggested for relevancy. To Alexey's point, it's difficult to
> navigate the more recent non-release blog posts (<1 year) because they are
> sprinkled in.
>
> A compromise for all of our points is to move the release notes to a
> separate section, and have a single section that's blog/news.
>
> And thanks for wanting to contribute, Aizhamal! Teaming up with you will
> make things run smoothly.
>
> On Tue, Mar 12, 2019 at 3:58 AM Alexey Romanenko 
> wrote:
>
>> +1 for “News” section. Though, I’d propose to exclude “release notes”
>> posts from “Blog” section list (since now we have quite regular releases,
>> it makes blog posts list not very readable for users) and move them to new
>> created News section. Also, this section could include the posts about
>> other Beam events, like meet-ups, conferences, project updates, etc. I’d
>> keep “Blog” section for more technical posts.
>>
>> On 12 Mar 2019, at 06:31, Pablo Estrada  wrote:
>>
>> I agree that the newsletter fits well as a blog post. I think that'd work
>> best.
>>
>> As for the cadence, I think quarterly would be a bit too infrequent. I
>> like once a month, or once every other month to have at least one per
>> release. Though happy to hear other people's thoughts.
>> Best
>> -P.
>>
>> On Mon, Mar 11, 2019 at 6:42 PM Thomas Weise  wrote:
>>
>>> +1 for single blog/news section
>>>
>>> Also I wouldn't mind quarterly cadence, to provide more focus for folks
>>> to contribute.
>>>
>>> On Mon, Mar 11, 2019 at 6:35 PM Kenneth Knowles  wrote:
>>>
 Could the newsletter be a blog entry? If you check out
 https://blogs.apache.org/ many of the posts are "Apache News
 Round-up". We could rename the "Blog" section to "News" if you ask me.

 Kenn

 On Mon, Mar 11, 2019 at 5:13 PM Aizhamal Nurmamat kyzy <
 aizha...@google.com> wrote:

> Hello everyone,
>
> I had a chat with Rose on how I can support the effort and keep
> sending the newsletters on a monthly basis. The new workflow would look as
> follows:
>
>1. Send out the same [CALL FOR ITEMS] where you can contribute to
>the Google doc with deadlines
>2. Edit the doc after the deadline
>3. Convert the file into Markdown
>4. Send in PR to add the file to Beam repo in Newsletter directory
>5. Have people send their fixes/updates through PRs
>
> In this effort, I can support Rose in steps 3 & 4.
>
> We would also need:
>
>- Create a Newsletter section under the Community tab
>- Write guidelines on newsletter contributions
>- Make a note about timing e.g. if upcoming event, then add to the
>next newsletter
>
> How does that sound to you all?
>
>
> On Wed, Mar 6, 2019 at 9:19 PM Rose Nguyen 
> wrote:
>
>> Good points. With the suggested workflow I think I can support a
>> quarterly newsletter. I'm also happy to get more involvement from others 
>> to
>> do this work and we can see what cadence that allows.
>>
>> On Wed, Mar 6, 2019 at 8:22 PM Kenneth Knowles 
>> wrote:
>>
>>> Good points Melissa & Austin.
>>>
>>>  - archive in the repo & on the website
>>>  - put missed items on the next newsletter, so anyone following sees
>>> them
>>>
>>> Kenn
>>>
>>> On Wed, Mar 6, 2019 at 3:26 PM Suneel Marthi 
>>> wrote:
>>>
 I believe there was also a Beam workshop or working session in
 Warsaw last week.

 On Wed, Mar 6, 2019 at 6:20 PM Austin Bennett <
 whatwouldausti...@gmail.com> wrote:

> +1 for archive in our repo.
>
> I do follow the newsletter, but am unlikely to go back and lo

Re: Google Season of Docs 2019

[Kenneth Knowles]

That's a lot of dev lists on one thread. Adding a couple more... :-)

Kenn

On Tue, Mar 12, 2019 at 7:17 PM Dave Fisher  wrote:

>
>
> > On Mar 12, 2019, at 7:01 PM, Huxing Zhang  wrote:
> >
> > Hi Dave,
> >
> > On Wed, Mar 13, 2019 at 2:14 AM Dave Fisher 
> wrote:
> >>
> >> Hi -
> >>
> >> Looks pretty cool.
> >
> > Yes, all the ASF projects could benefit from it. I think ASF should
> > apply it on behalf of all ASF projects, just like how the GSoC work.
> > Am I right?
>
> Yes! And the time is pretty quick. I’m hoping a someone over here will
> volunteer to help.
>
> Regards,
> Dave
>
>
> >
> >>
> >> Cc: to Apache Community Development.
> >>
> >> Regards,
> >> Dave
> >>
> >> Sent from my iPhone
> >>
> >>> On Mar 12, 2019, at 9:22 AM, Huxing Zhang  wrote:
> >>>
> >>> Hi,
> >>>
> >>> Google Season of Docs 2019[1] seems to be an interesting project,
> >>> which bring open source project and technical writer communities
> >>> together, just Like Google summer of code.
> >>>
> >>> I think the Dubbo can benefit from the project, especially the English
> >>> version of documentation could be improved.
> >>>
> >>> How do you think?
> >>>
> >>> [1] https://developers.google.com/season-of-docs/docs/timeline
> >>>
> >>> --
> >>> Best Regards！
> >>> Huxing
> >>
> >
> >
> > --
> > Best Regards！
> > Huxing
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>
>

Re: Looking for another reviewer and/or committer for beam metrics code.

[Pablo Estrada]

I've been looking at these recently, but I've opted to leave some stuff to
Robert. I'm happy to pick them back up if we're looking to reduce the load
on a single one : )
Best
-P.

On Tue, Mar 12, 2019 at 5:29 PM Alex Amato  wrote:

> Hi,
>
> Mikhail, Ryan and I have been working on some metric related PRs recently.
> Robert has been reviewing and committing these changes, but we would like
> another reviewer to get another opinion and reduce the volume for Robert.
>
> Ryan's goal is to implement the GetMetrics API
> 
>  and
> enable querying for metrics in the Flink Runner.
>
> Mikhail and I are finishing adding some standard metrics for the python
> SDK and the Dataflow Runner, and writing integration tests (SDK metrics
> design ):
>
>- Element Count
>- MeanByteCount
>- User Distribution Metrics
>
> Please let us know if you would be interested in learning more about
> metrics and get involved in reviewing the related PRs and we can add you as
> a reviewer.
>
> Relevant PRs;
> https://github.com/apache/beam/pull/7936
> https://github.com/apache/beam/pull/7995
> https://github.com/apache/beam/pull/8032
> https://github.com/apache/beam/pull/7971
> https://github.com/apache/beam/pull/7915
> https://github.com/apache/beam/pull/7899
>
>
>

Looking for another reviewer and/or committer for beam metrics code.

[Alex Amato]

Hi,

Mikhail, Ryan and I have been working on some metric related PRs recently.
Robert has been reviewing and committing these changes, but we would like
another reviewer to get another opinion and reduce the volume for Robert.

Ryan's goal is to implement the GetMetrics API

and
enable querying for metrics in the Flink Runner.

Mikhail and I are finishing adding some standard metrics for the python SDK
and the Dataflow Runner, and writing integration tests (SDK metrics design
):

   - Element Count
   - MeanByteCount
   - User Distribution Metrics

Please let us know if you would be interested in learning more about
metrics and get involved in reviewing the related PRs and we can add you as
a reviewer.

Relevant PRs;
https://github.com/apache/beam/pull/7936
https://github.com/apache/beam/pull/7995
https://github.com/apache/beam/pull/8032
https://github.com/apache/beam/pull/7971
https://github.com/apache/beam/pull/7915
https://github.com/apache/beam/pull/7899

Re: [ANNOUNCE] New committer announcement: Raghu Angadi

[Reuven Lax]

Congratulations!

On Mon, Mar 11, 2019 at 8:51 PM Raghu Angadi  wrote:

> Thank you all!
>
> On Mon, Mar 11, 2019 at 6:11 AM Maximilian Michels  wrote:
>
>> Congrats! :)
>>
>> On 11.03.19 14:01, Etienne Chauchot wrote:
>> > Congrats ! Well deserved
>> >
>> > Etienne
>> >
>> > Le lundi 11 mars 2019 à 13:22 +0100, Alexey Romanenko a écrit :
>> >> My congratulations, Raghu!
>> >>
>> >>> On 8 Mar 2019, at 10:39, Łukasz Gajowy > >>> > wrote:
>> >>>
>> >>> Congratulations! :)
>> >>>
>> >>> pt., 8 mar 2019 o 10:16 Gleb Kanterov > >>> > napisał(a):
>>  Congratulations!
>> 
>>  On Thu, Mar 7, 2019 at 11:52 PM Michael Luckey >  > wrote:
>> > Congrats Raghu!
>> >
>> > On Thu, Mar 7, 2019 at 8:06 PM Mark Liu > > > wrote:
>> >> Congrats!
>> >>
>> >> On Thu, Mar 7, 2019 at 10:45 AM Rui Wang > >> > wrote:
>> >>> Congrats Raghu!
>> >>>
>> >>>
>> >>> -Rui
>> >>>
>> >>> On Thu, Mar 7, 2019 at 10:22 AM Thomas Weise > >>> > wrote:
>>  Congrats!
>> 
>> 
>>  On Thu, Mar 7, 2019 at 10:11 AM Tim Robertson
>>  mailto:timrobertson...@gmail.com>>
>>  wrote:
>> > Congrats Raghu
>> >
>> > On Thu, Mar 7, 2019 at 7:09 PM Ahmet Altay > > > wrote:
>> >> Congratulations!
>> >>
>> >> On Thu, Mar 7, 2019 at 10:08 AM Ruoyun Huang
>> >> mailto:ruo...@google.com>> wrote:
>> >>> Thank you Raghu for your contribution!
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Mar 7, 2019 at 9:58 AM Connell O'Callaghan
>> >>> mailto:conne...@google.com>> wrote:
>>  Congratulation Raghu!!! Thank you for sharing Kenn!!!
>> 
>>  On Thu, Mar 7, 2019 at 9:55 AM Ismaël Mejía
>>  mailto:ieme...@gmail.com>> wrote:
>> > Congrats !
>> >
>> > Le jeu. 7 mars 2019 à 17:09, Aizhamal Nurmamat kyzy
>> > mailto:aizha...@google.com>> a
>> écrit :
>> >> Congratulations, Raghu!!!
>> >> On Thu, Mar 7, 2019 at 08:07 Kenneth Knowles
>> >> mailto:k...@apache.org>> wrote:
>> >>> Hi all,
>> >>>
>> >>> Please join me and the rest of the Beam PMC in welcoming
>> >>> a new committer: Raghu Angadi
>> >>>
>> >>> Raghu has been contributing to Beam since early 2016! He
>> >>> has continuously improved KafkaIO and supported on the
>> >>> user@ list but his community contributions are even more
>> >>> extensive, including reviews, dev@ list discussions,
>> >>> improvements and ideas across SqsIO, FileIO, PubsubIO,
>> >>> and the Dataflow and Samza runners. In consideration of
>> >>> Raghu's contributions, the Beam PMC trusts Raghu with the
>> >>> responsibilities of a Beam committer [1].
>> >>>
>> >>> Thank you, Raghu, for your contributions.
>> >>>
>> >>> Kenn
>> >>>
>> >>> [1]
>> >>>
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> >>> <
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> >
>> >>>
>> >>>
>> >>> --
>> >>> 
>> >>> Ruoyun  Huang
>> >>>
>> 
>> 
>>  --
>>  Cheers,
>>  Gleb
>> >>
>>
>

Re: Cross-language transform API

[Reuven Lax]

About schemas, pr/7952 adds schemas to the runner API.

This PR focuses on the Dataflow runner (not the portability runner).
However after some community discussion on the design (we are still free to
change these protos based on discussion), I would like to stabilize these
protos.

On Mon, Mar 11, 2019 at 11:04 AM Maximilian Michels  wrote:

> Thanks for the remarks. Correct, we do not need the static URN at all in
> the payload. We can pass the transform URN with the ExternalTransform as
> part of the ExpansionRequest. So this is sufficient for the Proto:
>
> message ConfigValue {
>string coder_urn = 1;
>bytes payload = 2;
> }
>
> message ExternalTransformPayload {
>map configuration = 1;
> }
>
>
> Considering Schemas, I'm not sure they are useful for the scope of the
> PR. I think basic Java Reflection is enough.
>
> Thanks,
> Max
>
> On 11.03.19 18:36, Robert Bradshaw wrote:
> > On Mon, Mar 11, 2019 at 6:05 PM Chamikara Jayalath 
> wrote:
> >>
> >> On Mon, Mar 11, 2019 at 9:27 AM Robert Bradshaw 
> wrote:
> >>>
> >>> On Mon, Mar 11, 2019 at 4:37 PM Maximilian Michels 
> wrote:
> 
> > Just to clarify. What's the reason for including a PROPERTIES enum
> here instead of directly making beam_urn a field of
> ExternalTransformPayload ?
> 
>  The URN is supposed to be static. We always use the same URN for this
>  type of external transform. We probably want an additional identifier
> to
>  point to the resource we want to configure.
> >>>
> >>> It does feel odd to not use the URN to specify the transform itself,
> >>> and embed the true identity in an inner proto. The notion of
> >>> "external" is just how it happens to be invoked in this pipeline, not
> >>> part of its intrinsic definition. As we want introspection
> >>> capabilities in the service, we should be able to use the URN at a top
> >>> level and know what kind of payload it expects. I would also like to
> >>> see this kind of information populated for non-extern transforms which
> >>> could be good for visibility (substitution, visualization, etc.) for
> >>> runners and other pipeline-consuming tools.
> >>>
>  Like so:
> 
>  message ExternalTransformPayload {
>  enum Enum {
>    PROPERTIES = 0
>    [(beam_urn) =
> "beam:external:transform:external_transform:v1"];
>  }
>  // A fully-qualified identifier, e.g. Java package + class
>  string identifier = 1;
> >>>
> >>> I'd rather the identifier have semantic rather than
> >>> implementation-specific meaning. e.g. one could imagine multiple
> >>> implementations of a given transform that different services could
> >>> offer.
> >>>
>  // the format may change to map if types are
> supported
>  map parameters = 2;
>  }
> 
>  The identifier could also be a URN.
> 
> > Can we change first version to map ? Otherwise the
> set of transforms we can support/test will be very limited.
> 
>  How do we do that? Do we define a set of standard coders for supported
>  types? On the Java side we can lookup the coder by extracting the
> field
>  from the Pojo, but we can't do that in Python.
> >>
> >>
> >> I'll let Reuven comment on exact relevance and timelines on Beam Schema
> related work here but till we have that probably we can support the
> standard set of coders that are well defined here ?
> >>
> https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L542
> >>
> >> So in Python side the ExternalTransform can take a list of parameters
> (of types that have standard coders) which will be converted to bytes to be
> sent over the wire. In Java side corresponding standard coders (which are
> determined by introspection of transform builder's payload POJO) can be
> used to covert bytes to objects.
> >
> > They also need to agree on the field types as well as names, so would
> > it be map>. I'm not sure the tradeoff
> > between going further down this road vs. getting schemas up to par in
> > Python (and, next, Go). And supporting this long term in parallel to
> > what we come up with schemas.
> >
> >> Hopefully Beam schema work will give us a more generalized way to
> convert objects across languages (for example, Python object -> Python Row
> + Schema -> Java Row + Schema -> Java object). Note that we run into the
> same issue when data tries to cross SDK boundaries when executing
> cross-language pipelines.
> >
> > +1, which is another reason I want to accelerate the language
> > independence of schemas.
> >
> > Can we re-use some of the Beam schemas-related work/utilities here ?
> 
>  Yes, that was the plan.
> >>>
> >>> On this note, Reuven, what is the plan (and timeline) for a
> >>> language-independent representation of schemas? The crux of the
> >>> problem is that the user needs to specify some kind of configuration
> >>> (call it C) to construct the transform (call it T). This would be
> >>> handled by 

Re: Build broken: repo.spring.io is down

[Mikhail Gryzykhin]

Thanks Kyle, that worked.

Does anyone know the reason why we declare same repositories in two
different locations?
Can we remove one of duplicates?

Regards,
--Mikhail

Have feedback ?


On Tue, Mar 12, 2019 at 1:14 PM Kyle Weaver  wrote:

> I commented out this line and it built fine:
> https://github.com/apache/beam/blob/c41e4fbbeb6ec622a0072e01afcba95428faafb9/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy#L44
>
> Kyle Weaver |  Software Engineer |  kcwea...@google.com |  +1650203
>
>
> On Tue, Mar 12, 2019 at 1:03 PM Mikhail Gryzykhin 
> wrote:
>
>> I tried to replace repo.sprint.io
>> 
>> with mavenCentral() that seem to have relevant plugin (propdeps-plugin
>> 
>> in my case), but gradle still fails to fetch it.
>>
>> Did anyone else had success? Build fails on my local machine as well.
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback ?
>>
>>
>> On Tue, Mar 12, 2019 at 11:25 AM Kyle Weaver  wrote:
>>
>>> Looks like this is still ongoing. Would greatly appreciate a fix if
>>> anyone's got one.
>>>
>>> Thanks,
>>> Kyle
>>>
>>> Kyle Weaver |  Software Engineer |  kcwea...@google.com |  +1650203
>>>
>>>
>>> On Tue, Mar 12, 2019 at 8:17 AM Maximilian Michels 
>>> wrote:
>>>
 FYI: Our build system is broken at the moment due to
 https://repo.spring.io being down.

 If this is not a temporary issue, we could try to switch to a different
 repository.

 16:07:02 FAILURE: Build failed with an exception.
 16:07:02
 16:07:02 * What went wrong:
 16:07:02 Execution failed for task ':beam-model-pipeline:compileJava'.
 16:07:02 > Could not resolve all files for configuration
 ':beam-model-pipeline:errorprone'.
 16:07:02> Could not resolve
 com.google.errorprone:error_prone_core:latest.release.
 16:07:02  Required by:
 16:07:02  project :beam-model-pipeline
 16:07:02   > Failed to list versions for
 com.google.errorprone:error_prone_core.
 16:07:02  > Unable to load Maven meta-data from

 https://repo.spring.io/plugins-release/com/google/errorprone/error_prone_core/maven-metadata.xml
 .
 16:07:02 > Could not HEAD
 '
 https://repo.spring.io/plugins-release/com/google/errorprone/error_prone_core/maven-metadata.xml
 '.
 16:07:02> Read timed out
 16:07:02

 https://builds.apache.org/job/beam_PreCommit_Java_Commit/4722/console

Re: Build broken: repo.spring.io is down

[Kyle Weaver]

I commented out this line and it built fine:
https://github.com/apache/beam/blob/c41e4fbbeb6ec622a0072e01afcba95428faafb9/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy#L44

Kyle Weaver |  Software Engineer |  kcwea...@google.com |  +1650203


On Tue, Mar 12, 2019 at 1:03 PM Mikhail Gryzykhin  wrote:

> I tried to replace repo.sprint.io
> 
> with mavenCentral() that seem to have relevant plugin (propdeps-plugin
> 
> in my case), but gradle still fails to fetch it.
>
> Did anyone else had success? Build fails on my local machine as well.
>
> Regards,
> --Mikhail
>
> Have feedback ?
>
>
> On Tue, Mar 12, 2019 at 11:25 AM Kyle Weaver  wrote:
>
>> Looks like this is still ongoing. Would greatly appreciate a fix if
>> anyone's got one.
>>
>> Thanks,
>> Kyle
>>
>> Kyle Weaver |  Software Engineer |  kcwea...@google.com |  +1650203
>>
>>
>> On Tue, Mar 12, 2019 at 8:17 AM Maximilian Michels 
>> wrote:
>>
>>> FYI: Our build system is broken at the moment due to
>>> https://repo.spring.io being down.
>>>
>>> If this is not a temporary issue, we could try to switch to a different
>>> repository.
>>>
>>> 16:07:02 FAILURE: Build failed with an exception.
>>> 16:07:02
>>> 16:07:02 * What went wrong:
>>> 16:07:02 Execution failed for task ':beam-model-pipeline:compileJava'.
>>> 16:07:02 > Could not resolve all files for configuration
>>> ':beam-model-pipeline:errorprone'.
>>> 16:07:02> Could not resolve
>>> com.google.errorprone:error_prone_core:latest.release.
>>> 16:07:02  Required by:
>>> 16:07:02  project :beam-model-pipeline
>>> 16:07:02   > Failed to list versions for
>>> com.google.errorprone:error_prone_core.
>>> 16:07:02  > Unable to load Maven meta-data from
>>>
>>> https://repo.spring.io/plugins-release/com/google/errorprone/error_prone_core/maven-metadata.xml
>>> .
>>> 16:07:02 > Could not HEAD
>>> '
>>> https://repo.spring.io/plugins-release/com/google/errorprone/error_prone_core/maven-metadata.xml
>>> '.
>>> 16:07:02> Read timed out
>>> 16:07:02
>>>
>>> https://builds.apache.org/job/beam_PreCommit_Java_Commit/4722/console
>>>
>>>

Re: Beam Meetups Feb 2019

[Austin Bennett]

Hi Teja and All,

The video recordings from the recent SF meetup have been posted to the Beam
YouTube channel (thanks, Matthias!).

Links:
General Beam YouTube:
https://www.youtube.com/channel/UChNnb_YO_7B0HlW6FhAXZZQ

*Beam Introduction*:  https://www.youtube.com/watch?v=Ao2NM8rvKZY
*TFX*:  https://www.youtube.com/watch?v=2MDEkn6_Mig
*Python Streaming Pipelines with Beam on Flink*:
https://www.youtube.com/watch?v=4jXQtt1McvM
*Dynamic Pricing of Lyft Rides using Streaming*:
https://www.youtube.com/watch?v=oQHyLfiv8Aw

Cheers,
Austin

On Mon, Feb 11, 2019 at 10:32 PM Teja MVSR  wrote:

> Hi,
>
> Can you please provide any video recordings if they are available?
>
> Thanks,
> Teja
>
> On Mon, Feb 11, 2019, 4:51 PM Austin Bennett  wrote:
>
>> The slides from Tyler's presentation found:
>> http://s.apache.org/beam-intro-feb-2019
>>
>> I'll also send out links to videos once I get my hands on them (@Mark
>> Grover  ).
>>
>> On Mon, Feb 11, 2019 at 9:48 AM Thomas Weise  wrote:
>>
>>> Here are slides for 2 of the presentations from the Lyft meetup:
>>>
>>> Python/Flink/Streaming: http://go.lyft.com/python-flink-beam-meetup-2019
>>> Use Case:
>>> https://www.slideshare.net/AmarPai2/dynamic-pricing-of-lyft-rides-using-streaming
>>>
>>> +Tyler Akidau  do you have pointers for the others
>>> by chance?
>>>
>>>
>>> On Fri, Feb 8, 2019 at 4:22 PM Kenneth Knowles  wrote:
>>>
 Yea, wow. 300 is huge! Nice. Looking forward to the Feb 21 meetup.

 Kenn

 On Fri, Feb 8, 2019 at 3:02 PM Matthias Baetens <
 baetensmatth...@gmail.com> wrote:

> Wow, that is awesome, Joana! Great job to everyone involved! :-)
>
> On Fri, 8 Feb 2019 at 22:42, Joana Filipa Bernardo Carrasqueira <
> joanafil...@google.com> wrote:
>
>> Hi all,
>>
>> I would like to take a moment to acknowledge the fact that yesterday
>> we had nearly 300 people at the Beam Meetup at Lyft!
>>
>> It was a remarkable event with great presentations and engagement
>> from the audience! It's great to see the community growing!
>>
>> [image: Lyft.jpg]
>>
>> For those in the Seattle area on Feb 21st, we will host another Beam
>> Meetup so help us spreading the word! Check the details here
>> .
>>
>> Have a great weekend!
>>
>> --
>>
>> *Joana Carrasqueira*
>>
>> Cloud Developer Relations Events Manager
>>
>> 415-602-2507 Mobile
>>
>> 1160 N Mathilda Ave, Sunnyvale, CA 94089
>>
>>
>>

Re: pylint command failing without an error? Blocked

[Alex Amato]

ahh, thanks somehow I missed that.

On Tue, Mar 12, 2019 at 11:32 AM Ahmet Altay  wrote:

> Error is in the logs you shared (from gradle scan)
>
> Running pycodestyle for module apache_beam  gen_protos.py  setup.py
> test_config.py:
> -> apache_beam/runners/dataflow/dataflow_metrics.py:49:1: E303 too many
> blank lines (4)
> -> apache_beam/runners/dataflow/dataflow_metrics.py:285:3: E303 too many
> blank lines (2)
> Command exited with non-zero status 1
>
> On Tue, Mar 12, 2019 at 11:27 AM Alex Amato  wrote:
>
>> Not sure how to proceed with my PR, this error says my code is 10/10 but
>> the command is still returning an error and fails.
>>
>> https://scans.gradle.com/s/47d4pkuf4tp46
>>
>>
>> :beam-sdks-python:lintPy27 FAILED
>> GLOB sdist-make: /usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/setup.py
>> py27-lint recreate: /usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/target/.tox/py27-lint
>> py27-lint installdeps: pycodestyle==2.3.1, pylint==1.9.3, future==0.16.0,
>> isort==4.2.15, flake8==3.5.0
>> WARNING:Discarding $PYTHONPATH from environment, to override specify
>> PYTHONPATH in 'passenv' in your configuration.
>> py27-lint inst: /usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/target/.tox/dist/apache-beam-2.12.0.dev0.zip
>> py27-lint installed: DEPRECATION: Python 2.7 will reach the end of its
>> life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't
>> be maintained after that date. A future version of pip will drop support
>> for Python
>> 2.7.,apache-beam==2.12.0.dev0,astroid==1.6.5,avro==1.8.2,backports.functools-lru-cache==1.5,certifi==2019.3.9,chardet==3.0.4,configparser==3.7.3,crcmod==1.7,dill==0.2.9,docopt==0.6.2,enum34==1.1.6,fastavro==0.21.19,flake8==3.5.0,funcsigs==1.0.2,future==0.16.0,futures==3.2.0,grpcio==1.19.0,hdfs==2.2.2,httplib2==0.11.3,idna==2.8,isort==4.2.15,lazy-object-proxy==1.3.1,mccabe==0.6.1,mock==2.0.0,monotonic==1.5,nose==1.3.7,numpy==1.16.2,oauth2client==3.0.0,pandas==0.23.4,parameterized==0.6.3,pbr==5.1.3,protobuf==3.7.0,pyarrow==0.11.1,pyasn1==0.4.5,pyasn1-modules==0.2.4,pycodestyle==2.3.1,pydot==1.2.4,pyflakes==1.6.0,PyHamcrest==1.9.0,pylint==1.9.3,pyparsing==2.3.1,python-dateutil==2.8.0,pytz==2018.9,PyVCF==0.6.8,PyYAML==3.13,requests==2.21.0,rsa==4.0,singledispatch==3.4.0.3,six==1.12.0,tenacity==5.0.3,typing==3.6.6,urllib3==1.24.1,wrapt==1.11.1
>> py27-lint runtests: PYTHONHASHSEED='4209602030'
>> py27-lint runtests: commands[0] | pylint --version
>> Using config file /usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/.pylintrc
>> pylint 1.9.3,
>> astroid 1.6.5
>> Python 2.7.14+ (default, Dec 5 2017, 15:17:02)
>> [GCC 7.2.0]
>> py27-lint runtests: commands[1] | pip --version
>> pip 19.0.3 from /usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/target/.tox/py27-lint/lib/python2.7/site-packages/pip
>> (python 2.7)
>> py27-lint runtests: commands[2] | /usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/scripts/run_tox_cleanup.sh
>> py27-lint runtests: commands[3] | time
>> /usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/scripts/run_pylint.sh
>> Skipping lint for generated files: bigquery_v2_client.py,
>> bigquery_v2_messages.py, dataflow_v1b3_client.py,
>> dataflow_v1b3_messages.py, storage_v1_client.py, storage_v1_messages.py,
>> proto2_coder_test_messages_pb2.py, beam_artifact_api_pb2_grpc.py,
>> beam_artifact_api_pb2.py, beam_expansion_api_pb2_grpc.py,
>> beam_expansion_api_pb2.py, beam_fn_api_pb2_grpc.py, beam_fn_api_pb2.py,
>> beam_job_api_pb2_grpc.py, beam_job_api_pb2.py,
>> beam_provision_api_pb2_grpc.py, beam_provision_api_pb2.py,
>> beam_runner_api_pb2_grpc.py, beam_runner_api_pb2.py, endpoints_pb2_grpc.py,
>> endpoints_pb2.py, metrics_pb2_grpc.py, metrics_pb2.py,
>> standard_window_fns_pb2_grpc.py, standard_window_fns_pb2.py
>> Running pylint for module apache_beam gen_protos.py setup.py
>> test_config.py:
>> Using config file /usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/.pylintrc
>> 
>> Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
>> Running pycodestyle for module apache_beam gen_protos.py setup.py
>> test_config.py:
>> apache_beam/runners/dataflow/dataflow_metrics.py:49:1: E303 too many
>> blank lines (4)
>> apache_beam/runners/dataflow/dataflow_metrics.py:285:3: E303 too many
>> blank lines (2)
>> Command exited with non-zero status 1
>> 499.83user 15.71system 1:36.48elapsed 534%CPU (0avgtext+0avgdata
>> 502996maxresident)k
>> 8inputs+224outputs (0major+816639minor)pagefaults 0swaps
>> ERROR: InvocationError for command '/usr/bin/time
>> /usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/scripts/run_pylint.sh' (exited with
>> code 1)
>> ___ summary
>> ___

Re: Build broken: repo.spring.io is down

[Mikhail Gryzykhin]

I tried to replace repo.sprint.io
 with
mavenCentral() that seem to have relevant plugin (propdeps-plugin

in my case), but gradle still fails to fetch it.

Did anyone else had success? Build fails on my local machine as well.

Regards,
--Mikhail

Have feedback ?


On Tue, Mar 12, 2019 at 11:25 AM Kyle Weaver  wrote:

> Looks like this is still ongoing. Would greatly appreciate a fix if
> anyone's got one.
>
> Thanks,
> Kyle
>
> Kyle Weaver |  Software Engineer |  kcwea...@google.com |  +1650203
>
>
> On Tue, Mar 12, 2019 at 8:17 AM Maximilian Michels  wrote:
>
>> FYI: Our build system is broken at the moment due to
>> https://repo.spring.io being down.
>>
>> If this is not a temporary issue, we could try to switch to a different
>> repository.
>>
>> 16:07:02 FAILURE: Build failed with an exception.
>> 16:07:02
>> 16:07:02 * What went wrong:
>> 16:07:02 Execution failed for task ':beam-model-pipeline:compileJava'.
>> 16:07:02 > Could not resolve all files for configuration
>> ':beam-model-pipeline:errorprone'.
>> 16:07:02> Could not resolve
>> com.google.errorprone:error_prone_core:latest.release.
>> 16:07:02  Required by:
>> 16:07:02  project :beam-model-pipeline
>> 16:07:02   > Failed to list versions for
>> com.google.errorprone:error_prone_core.
>> 16:07:02  > Unable to load Maven meta-data from
>>
>> https://repo.spring.io/plugins-release/com/google/errorprone/error_prone_core/maven-metadata.xml
>> .
>> 16:07:02 > Could not HEAD
>> '
>> https://repo.spring.io/plugins-release/com/google/errorprone/error_prone_core/maven-metadata.xml
>> '.
>> 16:07:02> Read timed out
>> 16:07:02
>>
>> https://builds.apache.org/job/beam_PreCommit_Java_Commit/4722/console
>>
>>

Re: pylint command failing without an error? Blocked

[Ahmet Altay]

Error is in the logs you shared (from gradle scan)

Running pycodestyle for module apache_beam  gen_protos.py  setup.py
test_config.py:
-> apache_beam/runners/dataflow/dataflow_metrics.py:49:1: E303 too many
blank lines (4)
-> apache_beam/runners/dataflow/dataflow_metrics.py:285:3: E303 too many
blank lines (2)
Command exited with non-zero status 1



On Tue, Mar 12, 2019 at 11:27 AM Alex Amato  wrote:

> Not sure how to proceed with my PR, this error says my code is 10/10 but
> the command is still returning an error and fails.
>
> https://scans.gradle.com/s/47d4pkuf4tp46
>
>
> :beam-sdks-python:lintPy27 FAILED
> GLOB sdist-make: /usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/setup.py
> py27-lint recreate: /usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/target/.tox/py27-lint
> py27-lint installdeps: pycodestyle==2.3.1, pylint==1.9.3, future==0.16.0,
> isort==4.2.15, flake8==3.5.0
> WARNING:Discarding $PYTHONPATH from environment, to override specify
> PYTHONPATH in 'passenv' in your configuration.
> py27-lint inst: /usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/target/.tox/dist/apache-beam-2.12.0.dev0.zip
> py27-lint installed: DEPRECATION: Python 2.7 will reach the end of its
> life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't
> be maintained after that date. A future version of pip will drop support
> for Python
> 2.7.,apache-beam==2.12.0.dev0,astroid==1.6.5,avro==1.8.2,backports.functools-lru-cache==1.5,certifi==2019.3.9,chardet==3.0.4,configparser==3.7.3,crcmod==1.7,dill==0.2.9,docopt==0.6.2,enum34==1.1.6,fastavro==0.21.19,flake8==3.5.0,funcsigs==1.0.2,future==0.16.0,futures==3.2.0,grpcio==1.19.0,hdfs==2.2.2,httplib2==0.11.3,idna==2.8,isort==4.2.15,lazy-object-proxy==1.3.1,mccabe==0.6.1,mock==2.0.0,monotonic==1.5,nose==1.3.7,numpy==1.16.2,oauth2client==3.0.0,pandas==0.23.4,parameterized==0.6.3,pbr==5.1.3,protobuf==3.7.0,pyarrow==0.11.1,pyasn1==0.4.5,pyasn1-modules==0.2.4,pycodestyle==2.3.1,pydot==1.2.4,pyflakes==1.6.0,PyHamcrest==1.9.0,pylint==1.9.3,pyparsing==2.3.1,python-dateutil==2.8.0,pytz==2018.9,PyVCF==0.6.8,PyYAML==3.13,requests==2.21.0,rsa==4.0,singledispatch==3.4.0.3,six==1.12.0,tenacity==5.0.3,typing==3.6.6,urllib3==1.24.1,wrapt==1.11.1
> py27-lint runtests: PYTHONHASHSEED='4209602030'
> py27-lint runtests: commands[0] | pylint --version
> Using config file /usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/.pylintrc
> pylint 1.9.3,
> astroid 1.6.5
> Python 2.7.14+ (default, Dec 5 2017, 15:17:02)
> [GCC 7.2.0]
> py27-lint runtests: commands[1] | pip --version
> pip 19.0.3 from /usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/target/.tox/py27-lint/lib/python2.7/site-packages/pip
> (python 2.7)
> py27-lint runtests: commands[2] | /usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/scripts/run_tox_cleanup.sh
> py27-lint runtests: commands[3] | time
> /usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/scripts/run_pylint.sh
> Skipping lint for generated files: bigquery_v2_client.py,
> bigquery_v2_messages.py, dataflow_v1b3_client.py,
> dataflow_v1b3_messages.py, storage_v1_client.py, storage_v1_messages.py,
> proto2_coder_test_messages_pb2.py, beam_artifact_api_pb2_grpc.py,
> beam_artifact_api_pb2.py, beam_expansion_api_pb2_grpc.py,
> beam_expansion_api_pb2.py, beam_fn_api_pb2_grpc.py, beam_fn_api_pb2.py,
> beam_job_api_pb2_grpc.py, beam_job_api_pb2.py,
> beam_provision_api_pb2_grpc.py, beam_provision_api_pb2.py,
> beam_runner_api_pb2_grpc.py, beam_runner_api_pb2.py, endpoints_pb2_grpc.py,
> endpoints_pb2.py, metrics_pb2_grpc.py, metrics_pb2.py,
> standard_window_fns_pb2_grpc.py, standard_window_fns_pb2.py
> Running pylint for module apache_beam gen_protos.py setup.py
> test_config.py:
> Using config file /usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/.pylintrc
> 
> Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
> Running pycodestyle for module apache_beam gen_protos.py setup.py
> test_config.py:
> apache_beam/runners/dataflow/dataflow_metrics.py:49:1: E303 too many blank
> lines (4)
> apache_beam/runners/dataflow/dataflow_metrics.py:285:3: E303 too many
> blank lines (2)
> Command exited with non-zero status 1
> 499.83user 15.71system 1:36.48elapsed 534%CPU (0avgtext+0avgdata
> 502996maxresident)k
> 8inputs+224outputs (0major+816639minor)pagefaults 0swaps
> ERROR: InvocationError for command '/usr/bin/time
> /usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/scripts/run_pylint.sh' (exited with
> code 1)
> ___ summary
> 
> ERROR: py27-lint: commands failed
>

pylint command failing without an error? Blocked

[Alex Amato]

Not sure how to proceed with my PR, this error says my code is 10/10 but
the command is still returning an error and fails.

https://scans.gradle.com/s/47d4pkuf4tp46


:beam-sdks-python:lintPy27 FAILED
GLOB sdist-make: /usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/setup.py
py27-lint recreate: /usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/target/.tox/py27-lint
py27-lint installdeps: pycodestyle==2.3.1, pylint==1.9.3, future==0.16.0,
isort==4.2.15, flake8==3.5.0
WARNING:Discarding $PYTHONPATH from environment, to override specify
PYTHONPATH in 'passenv' in your configuration.
py27-lint inst: /usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/target/.tox/dist/apache-beam-2.12.0.dev0.zip
py27-lint installed: DEPRECATION: Python 2.7 will reach the end of its life
on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be
maintained after that date. A future version of pip will drop support for
Python
2.7.,apache-beam==2.12.0.dev0,astroid==1.6.5,avro==1.8.2,backports.functools-lru-cache==1.5,certifi==2019.3.9,chardet==3.0.4,configparser==3.7.3,crcmod==1.7,dill==0.2.9,docopt==0.6.2,enum34==1.1.6,fastavro==0.21.19,flake8==3.5.0,funcsigs==1.0.2,future==0.16.0,futures==3.2.0,grpcio==1.19.0,hdfs==2.2.2,httplib2==0.11.3,idna==2.8,isort==4.2.15,lazy-object-proxy==1.3.1,mccabe==0.6.1,mock==2.0.0,monotonic==1.5,nose==1.3.7,numpy==1.16.2,oauth2client==3.0.0,pandas==0.23.4,parameterized==0.6.3,pbr==5.1.3,protobuf==3.7.0,pyarrow==0.11.1,pyasn1==0.4.5,pyasn1-modules==0.2.4,pycodestyle==2.3.1,pydot==1.2.4,pyflakes==1.6.0,PyHamcrest==1.9.0,pylint==1.9.3,pyparsing==2.3.1,python-dateutil==2.8.0,pytz==2018.9,PyVCF==0.6.8,PyYAML==3.13,requests==2.21.0,rsa==4.0,singledispatch==3.4.0.3,six==1.12.0,tenacity==5.0.3,typing==3.6.6,urllib3==1.24.1,wrapt==1.11.1
py27-lint runtests: PYTHONHASHSEED='4209602030'
py27-lint runtests: commands[0] | pylint --version
Using config file /usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/.pylintrc
pylint 1.9.3,
astroid 1.6.5
Python 2.7.14+ (default, Dec 5 2017, 15:17:02)
[GCC 7.2.0]
py27-lint runtests: commands[1] | pip --version
pip 19.0.3 from /usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/target/.tox/py27-lint/lib/python2.7/site-packages/pip
(python 2.7)
py27-lint runtests: commands[2] | /usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/scripts/run_tox_cleanup.sh
py27-lint runtests: commands[3] | time
/usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/scripts/run_pylint.sh
Skipping lint for generated files: bigquery_v2_client.py,
bigquery_v2_messages.py, dataflow_v1b3_client.py,
dataflow_v1b3_messages.py, storage_v1_client.py, storage_v1_messages.py,
proto2_coder_test_messages_pb2.py, beam_artifact_api_pb2_grpc.py,
beam_artifact_api_pb2.py, beam_expansion_api_pb2_grpc.py,
beam_expansion_api_pb2.py, beam_fn_api_pb2_grpc.py, beam_fn_api_pb2.py,
beam_job_api_pb2_grpc.py, beam_job_api_pb2.py,
beam_provision_api_pb2_grpc.py, beam_provision_api_pb2.py,
beam_runner_api_pb2_grpc.py, beam_runner_api_pb2.py, endpoints_pb2_grpc.py,
endpoints_pb2.py, metrics_pb2_grpc.py, metrics_pb2.py,
standard_window_fns_pb2_grpc.py, standard_window_fns_pb2.py
Running pylint for module apache_beam gen_protos.py setup.py test_config.py:
Using config file /usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/.pylintrc

Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
Running pycodestyle for module apache_beam gen_protos.py setup.py
test_config.py:
apache_beam/runners/dataflow/dataflow_metrics.py:49:1: E303 too many blank
lines (4)
apache_beam/runners/dataflow/dataflow_metrics.py:285:3: E303 too many blank
lines (2)
Command exited with non-zero status 1
499.83user 15.71system 1:36.48elapsed 534%CPU (0avgtext+0avgdata
502996maxresident)k
8inputs+224outputs (0major+816639minor)pagefaults 0swaps
ERROR: InvocationError for command '/usr/bin/time
/usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/scripts/run_pylint.sh' (exited with code
1)
___ summary

ERROR: py27-lint: commands failed

Re: Build broken: repo.spring.io is down

[Kyle Weaver]

Looks like this is still ongoing. Would greatly appreciate a fix if
anyone's got one.

Thanks,
Kyle

Kyle Weaver |  Software Engineer |  kcwea...@google.com |  +1650203


On Tue, Mar 12, 2019 at 8:17 AM Maximilian Michels  wrote:

> FYI: Our build system is broken at the moment due to
> https://repo.spring.io being down.
>
> If this is not a temporary issue, we could try to switch to a different
> repository.
>
> 16:07:02 FAILURE: Build failed with an exception.
> 16:07:02
> 16:07:02 * What went wrong:
> 16:07:02 Execution failed for task ':beam-model-pipeline:compileJava'.
> 16:07:02 > Could not resolve all files for configuration
> ':beam-model-pipeline:errorprone'.
> 16:07:02> Could not resolve
> com.google.errorprone:error_prone_core:latest.release.
> 16:07:02  Required by:
> 16:07:02  project :beam-model-pipeline
> 16:07:02   > Failed to list versions for
> com.google.errorprone:error_prone_core.
> 16:07:02  > Unable to load Maven meta-data from
>
> https://repo.spring.io/plugins-release/com/google/errorprone/error_prone_core/maven-metadata.xml
> .
> 16:07:02 > Could not HEAD
> '
> https://repo.spring.io/plugins-release/com/google/errorprone/error_prone_core/maven-metadata.xml
> '.
> 16:07:02> Read timed out
> 16:07:02
>
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/4722/console
>
>

Re: Always get to LGTM in Committer Guide

[Kenneth Knowles]

Indeed. I've opened a PR to record the change:
https://github.com/apache/beam/pull/8040

Kenn

On Tue, Mar 12, 2019 at 10:43 AM Rui Wang  wrote:

> Yes. I think we changed the policy in [1].
>
> Quote from the thread:
>
> (1) At least one committer is involved with the code review, as either a
> reviewer or as the author
> (2) A contributor has approved the change
>
> prior to merging any change.
>
> This changes our policy from its current requirement that at least one
> committer *who is not the author* has approved the change prior to merging.
> We believe that changing this process will improve code review throughput,
> reduce committer load, and engage more of the community in the code review
> process.
>
>
>
> [1]:
> https://lists.apache.org/thread.html/34a10b9d4d0b3b1cc92132779ab505bcb4a759aa9ae40f3338451d35@%3Cdev.beam.apache.org%3E
>
> -Rui
>
> On Tue, Mar 12, 2019 at 10:37 AM Reuven Lax  wrote:
>
>> You are right that it was decided that a contributor can review a
>> committer's PR. I think the committer guide was never updated, and we
>> should do so.
>>
>> On Tue, Mar 12, 2019 at 10:28 AM Gleb Kanterov  wrote:
>>
>>> Before pressing merge button I was familiarizing myself with committer
>>> guide [1]. It's saying:
>>>
>>> > A committer (who is not the author of the code) should signal this
>>> either by GitHub “approval” or by a comment such as “Looks good to me!”
>>> (LGTM). Any committer can then merge the pull request. It is fine for a
>>> committer to self-merge if another committer has reviewed the code and
>>> approved it, just be sure to be explicit about whose job it is!
>>>
>>> As I understand it, it's saying that the reviewer should be Beam
>>> Committer. However, I remember from my personal experience and reading "An
>>> approach to community building from Apache Beam" [2] that
>>>
>>> > either the reviewer or the author be a committer
>>>
>>> I'm wondering if we could rephrase our Commiter Guide a bit to make it
>>> clear.
>>>
>>> [1]: https://beam.apache.org/contribute/committer-guide/
>>> [2]:
>>> https://blogs.apache.org/comdev/entry/an-approach-to-community-building
>>>
>>>

Re: Always get to LGTM in Committer Guide

[Rui Wang]

Yes. I think we changed the policy in [1].

Quote from the thread:

(1) At least one committer is involved with the code review, as either a
reviewer or as the author
(2) A contributor has approved the change

prior to merging any change.

This changes our policy from its current requirement that at least one
committer *who is not the author* has approved the change prior to merging.
We believe that changing this process will improve code review throughput,
reduce committer load, and engage more of the community in the code review
process.



[1]:
https://lists.apache.org/thread.html/34a10b9d4d0b3b1cc92132779ab505bcb4a759aa9ae40f3338451d35@%3Cdev.beam.apache.org%3E

-Rui

On Tue, Mar 12, 2019 at 10:37 AM Reuven Lax  wrote:

> You are right that it was decided that a contributor can review a
> committer's PR. I think the committer guide was never updated, and we
> should do so.
>
> On Tue, Mar 12, 2019 at 10:28 AM Gleb Kanterov  wrote:
>
>> Before pressing merge button I was familiarizing myself with committer
>> guide [1]. It's saying:
>>
>> > A committer (who is not the author of the code) should signal this
>> either by GitHub “approval” or by a comment such as “Looks good to me!”
>> (LGTM). Any committer can then merge the pull request. It is fine for a
>> committer to self-merge if another committer has reviewed the code and
>> approved it, just be sure to be explicit about whose job it is!
>>
>> As I understand it, it's saying that the reviewer should be Beam
>> Committer. However, I remember from my personal experience and reading "An
>> approach to community building from Apache Beam" [2] that
>>
>> > either the reviewer or the author be a committer
>>
>> I'm wondering if we could rephrase our Commiter Guide a bit to make it
>> clear.
>>
>> [1]: https://beam.apache.org/contribute/committer-guide/
>> [2]:
>> https://blogs.apache.org/comdev/entry/an-approach-to-community-building
>>
>>

Re: Always get to LGTM in Committer Guide

[Reuven Lax]

You are right that it was decided that a contributor can review a
committer's PR. I think the committer guide was never updated, and we
should do so.

On Tue, Mar 12, 2019 at 10:28 AM Gleb Kanterov  wrote:

> Before pressing merge button I was familiarizing myself with committer
> guide [1]. It's saying:
>
> > A committer (who is not the author of the code) should signal this
> either by GitHub “approval” or by a comment such as “Looks good to me!”
> (LGTM). Any committer can then merge the pull request. It is fine for a
> committer to self-merge if another committer has reviewed the code and
> approved it, just be sure to be explicit about whose job it is!
>
> As I understand it, it's saying that the reviewer should be Beam
> Committer. However, I remember from my personal experience and reading "An
> approach to community building from Apache Beam" [2] that
>
> > either the reviewer or the author be a committer
>
> I'm wondering if we could rephrase our Commiter Guide a bit to make it
> clear.
>
> [1]: https://beam.apache.org/contribute/committer-guide/
> [2]:
> https://blogs.apache.org/comdev/entry/an-approach-to-community-building
>
>

Always get to LGTM in Committer Guide

[Gleb Kanterov]

Before pressing merge button I was familiarizing myself with committer
guide [1]. It's saying:

> A committer (who is not the author of the code) should signal this either
by GitHub “approval” or by a comment such as “Looks good to me!” (LGTM).
Any committer can then merge the pull request. It is fine for a committer
to self-merge if another committer has reviewed the code and approved it,
just be sure to be explicit about whose job it is!

As I understand it, it's saying that the reviewer should be Beam Committer.
However, I remember from my personal experience and reading "An approach to
community building from Apache Beam" [2] that

> either the reviewer or the author be a committer

I'm wondering if we could rephrase our Commiter Guide a bit to make it
clear.

[1]: https://beam.apache.org/contribute/committer-guide/
[2]: https://blogs.apache.org/comdev/entry/an-approach-to-community-building

Re: Apache Beam Newsletter - February/March 2019

[Rose Nguyen]

Once every 2 months works. We include both big and small items. It'll
provide the focus Thomas mentions but still catches the frequency that
Pablo suggested for relevancy. To Alexey's point, it's difficult to
navigate the more recent non-release blog posts (<1 year) because they are
sprinkled in.

A compromise for all of our points is to move the release notes to a
separate section, and have a single section that's blog/news.

And thanks for wanting to contribute, Aizhamal! Teaming up with you will
make things run smoothly.

On Tue, Mar 12, 2019 at 3:58 AM Alexey Romanenko 
wrote:

> +1 for “News” section. Though, I’d propose to exclude “release notes”
> posts from “Blog” section list (since now we have quite regular releases,
> it makes blog posts list not very readable for users) and move them to new
> created News section. Also, this section could include the posts about
> other Beam events, like meet-ups, conferences, project updates, etc. I’d
> keep “Blog” section for more technical posts.
>
> On 12 Mar 2019, at 06:31, Pablo Estrada  wrote:
>
> I agree that the newsletter fits well as a blog post. I think that'd work
> best.
>
> As for the cadence, I think quarterly would be a bit too infrequent. I
> like once a month, or once every other month to have at least one per
> release. Though happy to hear other people's thoughts.
> Best
> -P.
>
> On Mon, Mar 11, 2019 at 6:42 PM Thomas Weise  wrote:
>
>> +1 for single blog/news section
>>
>> Also I wouldn't mind quarterly cadence, to provide more focus for folks
>> to contribute.
>>
>> On Mon, Mar 11, 2019 at 6:35 PM Kenneth Knowles  wrote:
>>
>>> Could the newsletter be a blog entry? If you check out
>>> https://blogs.apache.org/ many of the posts are "Apache News Round-up".
>>> We could rename the "Blog" section to "News" if you ask me.
>>>
>>> Kenn
>>>
>>> On Mon, Mar 11, 2019 at 5:13 PM Aizhamal Nurmamat kyzy <
>>> aizha...@google.com> wrote:
>>>
 Hello everyone,

 I had a chat with Rose on how I can support the effort and keep sending
 the newsletters on a monthly basis. The new workflow would look as follows:

1. Send out the same [CALL FOR ITEMS] where you can contribute to
the Google doc with deadlines
2. Edit the doc after the deadline
3. Convert the file into Markdown
4. Send in PR to add the file to Beam repo in Newsletter directory
5. Have people send their fixes/updates through PRs

 In this effort, I can support Rose in steps 3 & 4.

 We would also need:

- Create a Newsletter section under the Community tab
- Write guidelines on newsletter contributions
- Make a note about timing e.g. if upcoming event, then add to the
next newsletter

 How does that sound to you all?


 On Wed, Mar 6, 2019 at 9:19 PM Rose Nguyen  wrote:

> Good points. With the suggested workflow I think I can support a
> quarterly newsletter. I'm also happy to get more involvement from others 
> to
> do this work and we can see what cadence that allows.
>
> On Wed, Mar 6, 2019 at 8:22 PM Kenneth Knowles 
> wrote:
>
>> Good points Melissa & Austin.
>>
>>  - archive in the repo & on the website
>>  - put missed items on the next newsletter, so anyone following sees
>> them
>>
>> Kenn
>>
>> On Wed, Mar 6, 2019 at 3:26 PM Suneel Marthi 
>> wrote:
>>
>>> I believe there was also a Beam workshop or working session in
>>> Warsaw last week.
>>>
>>> On Wed, Mar 6, 2019 at 6:20 PM Austin Bennett <
>>> whatwouldausti...@gmail.com> wrote:
>>>
 +1 for archive in our repo.

 I do follow the newsletter, but am unlikely to go back and look
 into the past for changes/updates.

 Would suggest that things that get missed in one newsletter (a
 concrete example, Suneel's talks not mentioned in the newsletter) 
 would get
 published in the next iteration, rather than editing the past 
 'published'
 newsletter.  Put another way, save editing the past for corrections 
 (typos,
 things being incorrect).  Else, I imagine that I'm unlikely to catch a
 great announcement that warranted being in the newsletter in the first
 place.  This certainly works better with a regular/frequent release
 cadence, like we arrived at for version releases (then, if something 
 misses
 one cut, it is not too big a deal, as the next release is coming soon).




 On Wed, Mar 6, 2019 at 12:50 PM Melissa Pashniak <
 meliss...@google.com> wrote:

>
> For step #2 (publishing onto the website), I think it would be
> good to stay consistent with our existing workflows if possible. 
> Rather
> than using an external tool, what about:
>
>

Re: hi from DevRel land

[Rose Nguyen]

Welcome, Reza! Really excited to have you!

On Tue, Mar 12, 2019 at 9:00 AM Reza Ardeshir Rokni 
wrote:

> Thanx folks!
>
> Oppsy on the link, here it is again:
>
> https://stackoverflow.com/questions/54422510/how-to-solve-duplicate-values-exception-when-i-create-pcollectionviewmapstring/54623618#54623618
>
> On Tue, 12 Mar 2019 at 23:32, Teja MVSR  wrote:
>
>> Hi Reza,
>>
>> I am also interested to contribute towards documentation. Please let me
>> know if I can be of any help.
>>
>> Thanks and Regards,
>> Teja.
>>
>> On Tue, Mar 12, 2019, 11:30 AM Kenneth Knowles  wrote:
>>
>>> This is great news.
>>>
>>> For the benefit of the list, I want to say how nice it has been when I
>>> have had a chance to work with you. I've learned a great deal real and
>>> complex use cases through those opportunities. I'm really excited that
>>> you'll be helping out Beam in this new role.
>>>
>>> Kenn
>>>
>>> On Tue, Mar 12, 2019 at 7:21 AM Valentyn Tymofieiev 
>>> wrote:
>>>
 Hi Reza!

 Welcome to Beam. Very nice to have you onboard. Btw, the link seems
 broken.

 Thanks,
 Valentyn

 On Tue, Mar 12, 2019 at 6:04 AM Reza Ardeshir Rokni 
 wrote:

> Hi Folks,
>
> Just wanted to say hi to the good folks in the Beam community in my
> new capacity as a Developer advocate for Beam/Dataflow @ Google. :-)
>
> At the moment I am working on a couple of blogs around the Timer and
> State API as well as some work on general patterns that I hope to
> contribute as documentation to the Beam site. An example of the patterns
> can be seen here:  LINK
>
> Hope to be adding many more in 2019 and really looking forward to
> being able to contribute to Beam in anyway that I can!
>
> Cheers
> Reza
>
>

-- 
Rose Thị Nguyễn

Re: hi from DevRel land

[Reza Ardeshir Rokni]

Thanx folks!

Oppsy on the link, here it is again:
https://stackoverflow.com/questions/54422510/how-to-solve-duplicate-values-exception-when-i-create-pcollectionviewmapstring/54623618#54623618

On Tue, 12 Mar 2019 at 23:32, Teja MVSR  wrote:

> Hi Reza,
>
> I am also interested to contribute towards documentation. Please let me
> know if I can be of any help.
>
> Thanks and Regards,
> Teja.
>
> On Tue, Mar 12, 2019, 11:30 AM Kenneth Knowles  wrote:
>
>> This is great news.
>>
>> For the benefit of the list, I want to say how nice it has been when I
>> have had a chance to work with you. I've learned a great deal real and
>> complex use cases through those opportunities. I'm really excited that
>> you'll be helping out Beam in this new role.
>>
>> Kenn
>>
>> On Tue, Mar 12, 2019 at 7:21 AM Valentyn Tymofieiev 
>> wrote:
>>
>>> Hi Reza!
>>>
>>> Welcome to Beam. Very nice to have you onboard. Btw, the link seems
>>> broken.
>>>
>>> Thanks,
>>> Valentyn
>>>
>>> On Tue, Mar 12, 2019 at 6:04 AM Reza Ardeshir Rokni 
>>> wrote:
>>>
 Hi Folks,

 Just wanted to say hi to the good folks in the Beam community in my new
 capacity as a Developer advocate for Beam/Dataflow @ Google. :-)

 At the moment I am working on a couple of blogs around the Timer and
 State API as well as some work on general patterns that I hope to
 contribute as documentation to the Beam site. An example of the patterns
 can be seen here:  LINK

 Hope to be adding many more in 2019 and really looking forward to being
 able to contribute to Beam in anyway that I can!

 Cheers
 Reza

Re: hi from DevRel land

[Teja MVSR]

Hi Reza,

I am also interested to contribute towards documentation. Please let me
know if I can be of any help.

Thanks and Regards,
Teja.

On Tue, Mar 12, 2019, 11:30 AM Kenneth Knowles  wrote:

> This is great news.
>
> For the benefit of the list, I want to say how nice it has been when I
> have had a chance to work with you. I've learned a great deal real and
> complex use cases through those opportunities. I'm really excited that
> you'll be helping out Beam in this new role.
>
> Kenn
>
> On Tue, Mar 12, 2019 at 7:21 AM Valentyn Tymofieiev 
> wrote:
>
>> Hi Reza!
>>
>> Welcome to Beam. Very nice to have you onboard. Btw, the link seems
>> broken.
>>
>> Thanks,
>> Valentyn
>>
>> On Tue, Mar 12, 2019 at 6:04 AM Reza Ardeshir Rokni 
>> wrote:
>>
>>> Hi Folks,
>>>
>>> Just wanted to say hi to the good folks in the Beam community in my new
>>> capacity as a Developer advocate for Beam/Dataflow @ Google. :-)
>>>
>>> At the moment I am working on a couple of blogs around the Timer and
>>> State API as well as some work on general patterns that I hope to
>>> contribute as documentation to the Beam site. An example of the patterns
>>> can be seen here:  LINK
>>>
>>> Hope to be adding many more in 2019 and really looking forward to being
>>> able to contribute to Beam in anyway that I can!
>>>
>>> Cheers
>>> Reza
>>>
>>>

Re: hi from DevRel land

[Kenneth Knowles]

This is great news.

For the benefit of the list, I want to say how nice it has been when I have
had a chance to work with you. I've learned a great deal real and complex
use cases through those opportunities. I'm really excited that you'll be
helping out Beam in this new role.

Kenn

On Tue, Mar 12, 2019 at 7:21 AM Valentyn Tymofieiev 
wrote:

> Hi Reza!
>
> Welcome to Beam. Very nice to have you onboard. Btw, the link seems broken.
>
> Thanks,
> Valentyn
>
> On Tue, Mar 12, 2019 at 6:04 AM Reza Ardeshir Rokni 
> wrote:
>
>> Hi Folks,
>>
>> Just wanted to say hi to the good folks in the Beam community in my new
>> capacity as a Developer advocate for Beam/Dataflow @ Google. :-)
>>
>> At the moment I am working on a couple of blogs around the Timer and
>> State API as well as some work on general patterns that I hope to
>> contribute as documentation to the Beam site. An example of the patterns
>> can be seen here:  LINK
>>
>> Hope to be adding many more in 2019 and really looking forward to being
>> able to contribute to Beam in anyway that I can!
>>
>> Cheers
>> Reza
>>
>>

Build broken: repo.spring.io is down

[Maximilian Michels]

FYI: Our build system is broken at the moment due to 
https://repo.spring.io being down.


If this is not a temporary issue, we could try to switch to a different 
repository.


16:07:02 FAILURE: Build failed with an exception.
16:07:02
16:07:02 * What went wrong:
16:07:02 Execution failed for task ':beam-model-pipeline:compileJava'.
16:07:02 > Could not resolve all files for configuration 
':beam-model-pipeline:errorprone'.
16:07:02    > Could not resolve 
com.google.errorprone:error_prone_core:latest.release.

16:07:02  Required by:
16:07:02  project :beam-model-pipeline
16:07:02   > Failed to list versions for 
com.google.errorprone:error_prone_core.
16:07:02  > Unable to load Maven meta-data from 
https://repo.spring.io/plugins-release/com/google/errorprone/error_prone_core/maven-metadata.xml.
16:07:02 > Could not HEAD 
'https://repo.spring.io/plugins-release/com/google/errorprone/error_prone_core/maven-metadata.xml'.

16:07:02    > Read timed out
16:07:02

https://builds.apache.org/job/beam_PreCommit_Java_Commit/4722/console

Re: hi from DevRel land

[Valentyn Tymofieiev]

Hi Reza!

Welcome to Beam. Very nice to have you onboard. Btw, the link seems broken.

Thanks,
Valentyn

On Tue, Mar 12, 2019 at 6:04 AM Reza Ardeshir Rokni 
wrote:

> Hi Folks,
>
> Just wanted to say hi to the good folks in the Beam community in my new
> capacity as a Developer advocate for Beam/Dataflow @ Google. :-)
>
> At the moment I am working on a couple of blogs around the Timer and State
> API as well as some work on general patterns that I hope to contribute as
> documentation to the Beam site. An example of the patterns can be seen
> here:  LINK
>
> Hope to be adding many more in 2019 and really looking forward to being
> able to contribute to Beam in anyway that I can!
>
> Cheers
> Reza
>
>

Re: hi from DevRel land

[Maximilian Michels]

Hi Reza,

Welcome! Sounds great. The documentation really needs an update in terms 
of timers/state.


Cheers,
Max

PS: The link is broken.

On 12.03.19 14:04, Reza Ardeshir Rokni wrote:

Hi Folks,

Just wanted to say hi to the good folks in the Beam community in my new 
capacity as a Developer advocate for Beam/Dataflow @ Google. :-)


At the moment I am working on a couple of blogs around the Timer and 
State API as well as some work on general patterns that I hope to 
contribute as documentation to the Beam site. An example of the patterns 
can be seen here: LINK 


Hope to be adding many more in 2019 and really looking forward to being 
able to contribute to Beam in anyway that I can!


Cheers
Reza

hi from DevRel land

[Reza Ardeshir Rokni]

Hi Folks,

Just wanted to say hi to the good folks in the Beam community in my new
capacity as a Developer advocate for Beam/Dataflow @ Google. :-)

At the moment I am working on a couple of blogs around the Timer and State
API as well as some work on general patterns that I hope to contribute as
documentation to the Beam site. An example of the patterns can be seen
here:  LINK

Hope to be adding many more in 2019 and really looking forward to being
able to contribute to Beam in anyway that I can!

Cheers
Reza

Re: Apache Beam Newsletter - February/March 2019

[Alexey Romanenko]

+1 for “News” section. Though, I’d propose to exclude “release notes” posts 
from “Blog” section list (since now we have quite regular releases, it makes 
blog posts list not very readable for users) and move them to new created News 
section. Also, this section could include the posts about other Beam events, 
like meet-ups, conferences, project updates, etc. I’d keep “Blog” section for 
more technical posts.

> On 12 Mar 2019, at 06:31, Pablo Estrada  wrote:
> 
> I agree that the newsletter fits well as a blog post. I think that'd work 
> best.
> 
> As for the cadence, I think quarterly would be a bit too infrequent. I like 
> once a month, or once every other month to have at least one per release. 
> Though happy to hear other people's thoughts.
> Best
> -P.
> 
> On Mon, Mar 11, 2019 at 6:42 PM Thomas Weise  > wrote:
> +1 for single blog/news section
> 
> Also I wouldn't mind quarterly cadence, to provide more focus for folks to 
> contribute.
> 
> On Mon, Mar 11, 2019 at 6:35 PM Kenneth Knowles  > wrote:
> Could the newsletter be a blog entry? If you check out 
> https://blogs.apache.org/  many of the posts are 
> "Apache News Round-up". We could rename the "Blog" section to "News" if you 
> ask me. 
> 
> Kenn
> 
> On Mon, Mar 11, 2019 at 5:13 PM Aizhamal Nurmamat kyzy  > wrote:
> Hello everyone,
> 
> I had a chat with Rose on how I can support the effort and keep sending the 
> newsletters on a monthly basis. The new workflow would look as follows:
> Send out the same [CALL FOR ITEMS] where you can contribute to the Google doc 
> with deadlines
> Edit the doc after the deadline
> Convert the file into Markdown
> Send in PR to add the file to Beam repo in Newsletter directory
> Have people send their fixes/updates through PRs
> In this effort, I can support Rose in steps 3 & 4.
> 
> We would also need:
> Create a Newsletter section under the Community tab
> Write guidelines on newsletter contributions
> Make a note about timing e.g. if upcoming event, then add to the next 
> newsletter
> How does that sound to you all?
> 
> 
> On Wed, Mar 6, 2019 at 9:19 PM Rose Nguyen  > wrote:
> Good points. With the suggested workflow I think I can support a quarterly 
> newsletter. I'm also happy to get more involvement from others to do this 
> work and we can see what cadence that allows.
> 
> On Wed, Mar 6, 2019 at 8:22 PM Kenneth Knowles  > wrote:
> Good points Melissa & Austin.
> 
>  - archive in the repo & on the website
>  - put missed items on the next newsletter, so anyone following sees them
> 
> Kenn
> 
> On Wed, Mar 6, 2019 at 3:26 PM Suneel Marthi  > wrote:
> I believe there was also a Beam workshop or working session in Warsaw last 
> week.
> 
> On Wed, Mar 6, 2019 at 6:20 PM Austin Bennett  > wrote:
> +1 for archive in our repo.  
> 
> I do follow the newsletter, but am unlikely to go back and look into the past 
> for changes/updates.  
> 
> Would suggest that things that get missed in one newsletter (a concrete 
> example, Suneel's talks not mentioned in the newsletter) would get published 
> in the next iteration, rather than editing the past 'published' newsletter.  
> Put another way, save editing the past for corrections (typos, things being 
> incorrect).  Else, I imagine that I'm unlikely to catch a great announcement 
> that warranted being in the newsletter in the first place.  This certainly 
> works better with a regular/frequent release cadence, like we arrived at for 
> version releases (then, if something misses one cut, it is not too big a 
> deal, as the next release is coming soon).  
> 
> 
> 
> 
> On Wed, Mar 6, 2019 at 12:50 PM Melissa Pashniak  > wrote:
> 
> For step #2 (publishing onto the website), I think it would be good to stay 
> consistent with our existing workflows if possible. Rather than using an 
> external tool, what about: 
> 
> After a google doc newsletter draft is ready, convert it into a standard 
> markdown file and put it into our GitHub repo, perhaps in a new newsletter 
> directory in the website community directory [1]. These would be listed for 
> browsing on a Newsletters page as mentioned in step #4. People can then just 
> open a PR to add missing things to the pages later, and the newsletter will 
> be automatically updated on the website through our standard website 
> workflow. It also avoids the potential issue of the source google docs 
> disappearing in the future, as they are stored in a community location.
> 
> [1] https://github.com/apache/beam/tree/master/website/src/community 
> 
> 
> 
> On Wed, Mar 6, 2019 at 10:36 AM Rose Nguyen  > wrote:
> I think that would be a great idea to change formats to help with

Re: JIRA hygiene

[Etienne Chauchot]

Hi Thomas,
I agree, the committer that merges a PR should close the ticket. And, if 
needed, he could discuss with the author
(inside the PR) to assess if the PR covers the ticket scope.
This is the rule I apply to myself when I merge a PR (even thought it has 
happened that I forgot to close one or two
tickets :) ) .
Etienne

Le lundi 11 mars 2019 à 14:17 -0700, Thomas Weise a écrit :
> JIRA probably deserves a separate discussion. It is messy.. We also have 
> examples of tickets being referenced by users
> that were not closed, although the feature long implemented or issue fixed.
> 
> There is no clear ownership in our workflow.
> 
> A while ago I proposed in another context to make resolving JIRA part of 
> committer duty. I would like to bring this up
> for discussion again:
> 
> https://github.com/apache/beam/pull/7129#discussion_r236405202
> 
> Thomas
> 
> 
> On Mon, Mar 11, 2019 at 1:47 PM Ahmet Altay  wrote:
> > I agree this is a good idea. I used the same technique for 2.11 blog post 
> > (JIRA release notes -> editorialized list
> > + diffed the dependencies).
> > 
> > On Mon, Mar 11, 2019 at 1:40 PM Kenneth Knowles  wrote:
> > > That is a good idea. The blog post is probably the main avenue where 
> > > folks will find out about new features or big
> > > fixes.
> > > When I did 2.10.0 I just used the automated Jira release notes and pulled 
> > > out significant things based on my
> > > judgment. I would also suggest that our Jira hygiene could be 
> > > significantly improved to make this process more
> > > effective.
> > > 
> > 
> > +1 to improving JIRA notes as well. Often times issues are closed with no 
> > real comments on what happened, has it
> > been resolved or not. It becomes an exercise on reading the linked PRs to 
> > figure out what happened.
> >  
> > > Kenn
> > > On Mon, Mar 11, 2019 at 1:04 PM Thomas Weise  wrote:
> > > > Ahmet, thanks managing the release!
> > > > I have a suggestion (not specific to only this release): 
> > > > 
> > > > The release blogs could be more useful to users. In this case, we have 
> > > > a long list of dependency updates on the
> > > > top, but probably the improvements and features section should come 
> > > > first. I was also very surprised to find
> > > > "Portable Flink runner support for running cross-language transforms." 
> > > > mentioned, since that is only being
> > > > worked on now. On the other hand, there are probably items that we miss.
> > > > 
> > > > Since this can only be addressed by more eyes, I suggest that going 
> > > > forward the blog pull request is included
> > > > and reviewed as part of the release vote.
> > > > 
> > > > Also, we should make announcing the release on Twitter part of the 
> > > > process.
> > 
> > This is actually part of the release process 
> > (https://beam.apache.org/contribute/release-guide/#social-media). I
> > missed it for 2.11. I will send an announcement on Twitter shortly. 
> > > > Thanks,
> > > > Thomas
> > > > 
> > > > 
> > > > On Mon, Mar 11, 2019 at 10:46 AM Ahmet Altay  wrote:
> > > > > I updated the JIRAs for these two PRs to set the fix version 
> > > > > correctly as 2.12.0. That should fix the release
> > > > > notes issue.
> > > > > 
> > > > > On Mon, Mar 11, 2019 at 10:44 AM Ahmet Altay  wrote:
> > > > > > Hi Etienne,
> > > > > > 
> > > > > > I cut the release branch on 2/14 at [1] (on Feb 14, 2019, 3:52 PM 
> > > > > > PST -- github timestamp). Release tag, as
> > > > > > you pointed out, points to a commit on Feb 25, 2019 11:48 PM PST. 
> > > > > > And that is a commit on the release
> > > > > > branch. 
> > > > > > 
> > > > > > After cutting the release branch, I only merged cherry picks from 
> > > > > > master to the release branch if a JIRA was
> > > > > > tagged as a release blocker and there was a PR to fix that specific 
> > > > > > issue. In case of these two PRs, they
> > > > > > were merged at Feb 20 and Feb 18 respectively. They were not 
> > > > > > included in the branch cut and I did not cherry
> > > > > > pick them either. I apologize if I missed a request to cherry pick 
> > > > > > these PRs.
> > > > > > 
> > > > > > Does this answer your question?
> > > > > > 
> > > > > > Ahmet
> > > > > > 
> > > > > > [1] 
> > > > > > https://github.com/apache/beam/commit/a103edafba569b2fd185b79adffd91aaacb790f0
> > > > > > On Mon, Mar 11, 2019 at 1:50 AM Etienne Chauchot 
> > > > > >  wrote:
> > > > > > > @Ahmet sorry I did not have time to check 2.11 release but a 
> > > > > > > fellow Beam contributor drew my attention on
> > > > > > > something:
> > > > > > > the 2.11 release tag points on commit of 02/26 and this[1] PR was 
> > > > > > > merged 02/20 and that [2] PR was merged
> > > > > > > on 02/18. So, both commits should be in the released code but 
> > > > > > > they are not. 
> > > > > > > [1] https://github.com/apache/beam/pull/7348[2] 
> > > > > > > https://github.com/apache/beam/pull/7751
> > > > > > > So at least for those 2 features the release notes do not co

Re: [VOTE] Release 2.11.0, release candidate #2

[Etienne Chauchot]

Ahmet, Yes  in a comment in this ticket 
https://issues.apache.org/jira/browse/BEAM-6292  I supposed there were cherry
picks.Thanks for the confirmation !
No problem about not having cherry picked these PRs they were not release 
blockers. It is just that these features were
announced for 2.11 in the release notes. My bad, I did not know the release cut 
date of 02/14 I should have targeted
them to 2.12 then.
Thanks
Etienne
Le lundi 11 mars 2019 à 10:44 -0700, Ahmet Altay a écrit :
> Hi Etienne,
> 
> I cut the release branch on 2/14 at [1] (on Feb 14, 2019, 3:52 PM PST -- 
> github timestamp). Release tag, as you
> pointed out, points to a commit on Feb 25, 2019 11:48 PM PST. And that is a 
> commit on the release branch. 
> 
> After cutting the release branch, I only merged cherry picks from master to 
> the release branch if a JIRA was tagged as
> a release blocker and there was a PR to fix that specific issue. In case of 
> these two PRs, they were merged at Feb 20
> and Feb 18 respectively. They were not included in the branch cut and I did 
> not cherry pick them either. I apologize
> if I missed a request to cherry pick these PRs.
> 
> Does this answer your question?
> 
> Ahmet
> 
> [1] 
> https://github.com/apache/beam/commit/a103edafba569b2fd185b79adffd91aaacb790f0
> On Mon, Mar 11, 2019 at 1:50 AM Etienne Chauchot  wrote:
> > @Ahmet sorry I did not have time to check 2.11 release but a fellow Beam 
> > contributor drew my attention on something:
> > the 2.11 release tag points on commit of 02/26 and this[1] PR was merged 
> > 02/20 and that [2] PR was merged on 02/18.
> > So, both commits should be in the released code but they are not. 
> > [1] https://github.com/apache/beam/pull/7348[2] 
> > https://github.com/apache/beam/pull/7751
> > So at least for those 2 features the release notes do not comply to content 
> > of the release. Is there a real problem
> > or did I miss something ?
> > Etienne
> > Le lundi 04 mars 2019 à 11:42 -0800, Ahmet Altay a écrit :
> > > Thank you for the additional votes and validations.
> > > Update: Binaries are pushed. Website updates are blocked on an issue that 
> > > is preventing beam-site changes to be
> > > synced the beam website. (INFRA-17953). I am waiting for that to be 
> > > resolved before sending an announcement.
> > > On Mon, Mar 4, 2019 at 3:00 AM Robert Bradshaw  
> > > wrote:
> > > > I see the vote has passed, but +1 (binding) from me as well.
> > > > 
> > > > 
> > > > 
> > > > On Mon, Mar 4, 2019 at 11:51 AM Jean-Baptiste Onofré 
> > > >  wrote:
> > > > 
> > > > >
> > > > 
> > > > > +1 (binding)
> > > > 
> > > > >
> > > > 
> > > > > Tested with beam-samples.
> > > > 
> > > > >
> > > > 
> > > > > Regards
> > > > 
> > > > > JB
> > > > 
> > > > >
> > > > 
> > > > > On 26/02/2019 10:40, Ahmet Altay wrote:
> > > > 
> > > > > > Hi everyone,
> > > > 
> > > > > >
> > > > 
> > > > > > Please review and vote on the release candidate #2 for the version
> > > > 
> > > > > > 2.11.0, as follows:
> > > > 
> > > > > >
> > > > 
> > > > > > [ ] +1, Approve the release
> > > > 
> > > > > > [ ] -1, Do not approve the release (please provide specific 
> > > > > > comments)
> > > > 
> > > > > >
> > > > 
> > > > > > The complete staging area is available for your review, which 
> > > > > > includes:
> > > > 
> > > > > > * JIRA release notes [1],
> > > > 
> > > > > > * the official Apache source release to be deployed to 
> > > > > > dist.apache.org
> > > > 
> > > > > >  [2], which is signed with the key with
> > > > 
> > > > > > fingerprint 64B84A5AD91F9C20F5E9D9A7D62E71416096FA00 [3],
> > > > 
> > > > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > 
> > > > > > * source code tag "v2.11.0-RC2" [5],
> > > > 
> > > > > > * website pull request listing the release [6] and publishing the 
> > > > > > API
> > > > 
> > > > > > reference manual [7].
> > > > 
> > > > > > * Python artifacts are deployed along with the source release to the
> > > > 
> > > > > > dist.apache.org  [2].
> > > > 
> > > > > > * Validation sheet with a tab for 2.11.0 release to help with 
> > > > > > validation
> > > > 
> > > > > > [8].
> > > > 
> > > > > >
> > > > 
> > > > > > The vote will be open for at least 72 hours. It is adopted by 
> > > > > > majority
> > > > 
> > > > > > approval, with at least 3 PMC affirmative votes.
> > > > 
> > > > > >
> > > > 
> > > > > > Thanks,
> > > > 
> > > > > > Ahmet
> > > > 
> > > > > >
> > > > 
> > > > > > [1]
> > > > 
> > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12344775
> > > > 
> > > > > > [2] https://dist.apache.org/repos/dist/dev/beam/2.11.0/
> > > > 
> > > > > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > > > 
> > > > > > [4] 
> > > > > > https://repository.apache.org/content/repositories/orgapachebeam-1064/
> > > > 
> > > > > > [5] https://github.com/apache/beam/tree/v2.11.0-RC2
> > > > 
> > > > > > [6] https://gi