Re: [DISCUSS] [DOC] Triggering is for sinks!

[Frances Perry]

I love this idea -- long live the separation of business logic and runtime
requirements.

On Thu, Nov 30, 2017 at 12:06 PM, Kenneth Knowles  wrote:

> Hi all,
>
> Triggers are one of the more novel aspects of Beam's support for unbounded
> data. They are also one of the most challenging aspects of the model.
>
> Ben & I have been working on a major new idea for how triggers could work
> in the Beam model. We think it will make triggers much more usable, create
> new opportunities for no-knobs execution/optimization, and improve
> compatibility with DSLs like SQL. (also eliminate a whole class of bugs)
>
> Triggering is for sinks!
>
> https://s.apache.org/beam-sink-triggers
>
> Please take a look at this "1"-pager and give feedback.
>
> Kenn
>

Re: Congratulations Davor!

[Frances Perry]

Woohoo! So well deserved.

On Thu, May 4, 2017 at 8:18 AM, Etienne Chauchot 
wrote:

> Congratulations Davor!
>
> Well deserved indeed!
>
>
>
> Le 04/05/2017 à 17:02, Thomas Groh a écrit :
>
>> Congratulations!
>>
>> On Thu, May 4, 2017 at 7:56 AM, Thomas Weise  wrote:
>>
>> Congrats!
>>>
>>>
>>> On Thu, May 4, 2017 at 7:53 AM, Sourabh Bajaj <
>>> sourabhba...@google.com.invalid> wrote:
>>>
>>> Congrats!!
 On Thu, May 4, 2017 at 7:48 AM Mingmin Xu  wrote:

 Congratulations @Davor!
>
>
> On May 4, 2017, at 7:08 AM, Amit Sela  wrote:
>>
>> Congratulations Davor!
>>
>> On Thu, May 4, 2017, 10:02 JingsongLee 
>>>
>> wrote:

> Congratulations!
>>> --
>>> From:Jesse Anderson 
>>> Time:2017 May 4 (Thu) 21:36
>>> To:dev 
>>> Subject:Re: Congratulations Davor!
>>> Congrats!
>>>
>>> On Thu, May 4, 2017, 6:20 AM Aljoscha Krettek >>>
>>> wrote:
>
>> Congrats! :-)

> On 4. May 2017, at 14:34, Kenneth Knowles 
 wrote:

> Awesome!
>
> On Thu, May 4, 2017 at 1:19 AM, Ted Yu 
>>
> wrote:

> Congratulations, Davor!
>>
>> On Thu, May 4, 2017 at 12:45 AM, Aviem Zur >
> wrote:

> Congrats Davor! :)
>>>
>>> On Thu, May 4, 2017 at 10:42 AM Jean-Baptiste Onofré <
>>>
>> j...@nanthrax.net>
>>>
 wrote:
>>>
>>> Congrats ! Well deserved ;)

 Regards
 JB

 On 05/04/2017 09:30 AM, Jason Kuster wrote:
> Hi all,
>
> The ASF has just published a blog post[1] welcoming new
>
 members
>>>
 of

> the
>>
>>> Apache Software Foundation, and our own Davor Bonaci is among
>
 them!
>
>> Congratulations and thank you to Davor for all of your work
>
 for
>>>
 the
>
>> Beam
>>>
 community, and the ASF at large. Well deserved.
>
> Best,
>
> Jason
>
> [1] https://blogs.apache.org/foundation/entry/the-apache-sof
> tware-foundation-welcomes
>
> P.S. I dug through the list to make sure I wasn't missing any
>
 other
>
>> Beam
>>>
 community members; if I have, my sincerest apologies and
>
 please
>>>
 recognize
>>>
 them on this or a new thread.
>
> --
 Jean-Baptiste Onofré
 jbono...@apache.org
 http://blog.nanthrax.net
 Talend - http://www.talend.com

 --

>>> Thanks,
>>>
>>> Jesse
>>>
>>>
>>>
>

Re: Help needed with WordCount commands

[Frances Perry]

This is a great way for folks to contribute as part of the community
hackathon.

I love the way these walkthroughs have instructions for all runners --
really helps drive home the portability message ;-)


On Fri, Apr 21, 2017 at 4:13 PM, Hadar Hod 
wrote:

> Hi everyone!
>
> Your help is needed with the WordCount [1] documentation!
>
> I'm a technical writer, continuing to work on Beam docs. In getting the
> site ready for the 2.0 release, I added a “How to run” section to each of
> the examples in the WordCount doc (staged here [2]). To be more exact, I
> added a Java section and Python section for each example (MinimalWC, WC,
> etc). Each section like this is broken down into the different runner
> options so that users can toggle to the one they need.
>
> *However(!)* the actual commands I added in the doc are guesses and have
> not been tested for correctness. I’d like to ask for your help in finishing
> up this page; If you know the correct directions for any of the runners /
> examples / SDKs, please leave me a comment in PR 222 [3] that I have out.
>
> Your help is greatly appreciated!
>
> Thank you,
> Hadar
>
> [1] https://beam.apache.org/get-started/wordcount-example/
> [2] *http://apache-beam-website-pull-requests.storage.
> googleapis.com/222/get-started/wordcount-example/index.html
>  started/wordcount-example/index.html>*
> [3] https://github.com/apache/beam-site/pull/222
>

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

[Frances Perry]

Woohoo! Congrats ;-)

On Thu, Jan 26, 2017 at 9:05 PM, Jean-Baptiste Onofré 
wrote:

> Welcome aboard !⁣
>
> Regards
> JB
>
> On Jan 27, 2017, 01:27, at 01:27, Davor Bonaci  wrote:
> >Please join me and the rest of Beam PMC in welcoming the following
> >contributors as our newest committers. They have significantly
> >contributed
> >to the project in different ways, and we look forward to many more
> >contributions in the future.
> >
> >* Stas Levin
> >Stas has contributed across the breadth of the project, from the Spark
> >runner to the core pieces and Java SDK. Looking at code contributions
> >alone, he authored 43 commits and reported 25 issues. Stas is very
> >active
> >on the mailing lists too, contributing to good discussions and
> >proposing
> >improvements to the Beam model.
> >
> >* Ahmet Altay
> >Ahmet is a major contributor to the Python SDK, both in terms of design
> >and
> >code contribution. Looking at code contributions alone, he authored 98
> >commits and reviewed dozens of pull requests. With Python SDK’s
> >imminent
> >merge to the master branch, Ahmet contributed towards establishing a
> >new
> >major component in Beam.
> >
> >* Pei He
> >Pei has been contributing to Beam since its inception, accumulating a
> >total
> >of 118 commits since February. He has made several major contributions,
> >most recently by redesigning IOChannelFactory / FileSystem APIs (in
> >progress), which would extend Beam’s portability to many additional
> >file
> >systems and cloud providers.
> >
> >Congratulations to all three! Welcome!
> >
> >Davor
>

Re: [VOTE] Merge Python SDK to the master branch

[Frances Perry]

[X] +1, Merge python-sdk branch to master after the 0.5.0 release, and
release it in the subsequent minor release

On Fri, Jan 20, 2017 at 9:06 AM, Davor Bonaci  wrote:

> [X] +1, Merge python-sdk branch to master after the 0.5.0 release, and
> release it in the subsequent minor release.
>
>
> On Fri, Jan 20, 2017 at 9:03 AM, Ahmet Altay 
> wrote:
>
> > Hi all,
> >
> >
> > Please review the earlier discussion on the status of the Python SDK [1]
> > and vote on merging the python-sdk branch to the master branch, as
> follows:
> >
> > [ ] +1, Merge python-sdk branch to master after the 0.5.0 release, and
> > release it in the subsequent minor release.
> >
> > [ ] -1, Continue development in python-sdk branch (please provide
> specific
> > comments).
> >
> > The vote will be open for at least 72 hours. This is a procedural vote;
> it
> > is adopted by majority approval of qualified votes with no minimums [2].
> >
> > Thank you,
> >
> > Ahmet
> >
> > [1]
> > https://lists.apache.org/thread.html/84a36cf0ad95a76e6bc444603ae87e
> > 7312023bc167a6ff3c57a956f1@%3Cdev.beam.apache.org%3E
> > [2] http://apache.org/foundation/voting.html
> >
>

Re: [DISCUSS] Python SDK status and next steps

[Frances Perry]

+1 merged after 0.5.

It's on a great trajectory in terms of development and community.

On Tue, Jan 17, 2017 at 5:48 PM, Kenneth Knowles 
wrote:

> Seems reasonable, and the timeline Davor suggests makes a lot of sense.
>
> On Tue, Jan 17, 2017 at 3:59 PM, Lukasz Cwik 
> wrote:
>
> > I'm also for merging to master.
> >
> > On Tue, Jan 17, 2017 at 3:39 PM, Jean-Baptiste Onofré 
> > wrote:
> >
> > > It makes sense to merge after 0.5.0 release.
> > >
> > > Good point Davor: +1
> > >
> > > Regards
> > > JB
> > >
> > >
> > > On 01/17/2017 03:34 PM, Davor Bonaci wrote:
> > >
> > >> +1. I think merging to master would be an awesome next step for the
> > Python
> > >> SDK.
> > >>
> > >> And, thanks for a great summary of the current state, roadmap, and
> > impact
> > >> to the project as a whole -- awesome!
> > >>
> > >> Process-wise, I'd suggest starting a formal vote once this discussion
> > >> seems
> > >> to be trending towards a conclusion, and complete the merge as soon as
> > the
> > >> next release (0.5.0) is cut. This would enable additional time before
> > >> 0.6.0
> > >> to figure out compliance, release process impact, etc.
> > >>
> > >> Great work everyone!
> > >>
> > >> On Tue, Jan 17, 2017 at 8:26 AM, Jean-Baptiste Onofré <
> j...@nanthrax.net>
> > >> wrote:
> > >>
> > >> Hi
> > >>>
> > >>> I didn't try the Python SDK recently but you provided a clear "state
> of
> > >>> the art". Anyway I'm in favor of merging things as quick as possible
> > >>> (assuming it's in a good shape in term of build, test, ...): it would
> > >>> potentially grow up the "external" contributions.
> > >>>
> > >>> So +1 from my side.
> > >>>
> > >>> Regards
> > >>> JB⁣​
> > >>>
> > >>> On Jan 17, 2017, 08:22, at 08:22, Ahmet Altay
>  > >
> > >>> wrote:
> > >>>
> >  Hi all,
> > 
> >  tl;dr: I would like to start a discussion about merging python-sdk
> >  branch
> >  to master branch. Python SDK is mature enough and merging it to
> master
> >  will
> >  accelerate its development and adoption.
> > 
> >  With a great effort from a lot of contributors(*), Python SDK [1] is
> >  now a
> >  mostly complete, tested, performant Python implementation of the
> Beam
> >  model. Since June, when we first started with Python SDK in Apache
> > Beam
> >  we
> >  have been continuously improving it.
> > 
> >  ** Python SDK currently supports:
> > 
> >  * Model: All main concepts are present (ParDo, GroupByKey, Windowing
> >  etc.).
> >  * IO: There are extensible APIs for writing new bounded sources and
> >  sinks.
> >  Implementations are provided for Text, Avro, BigQuery, and
> Datastore.
> >  * Runners: Python SDK has an extensible base runner module that
> allows
> >  building specific runners on top of it. The SDK comes with two
> > pipeline
> >  runners: DirectRunner and DataflowRunner; and it is possible to add
> >  more.
> >  The existing runners are currently limited to bounded execution and
> >  otherwise equivalent to their Java SDK counterparts in
> functionality.
> >  * Testing: Python SDK implements ValidatesRunner test framework for
> >  implementing integration test for current and future runners. There
> is
> >  unit
> >  test coverage for all modules, and a number of integrations test for
> >  validating existing runners.
> >  * Documentation and examples: Documentation work has started on
> Python
> >  SDK.
> >  Beam Programming Guide page has been updated to include Python [2].
> > The
> >  code comes with many ready to use examples and we are in a good
> place
> >  to
> >  start documenting those on the website.
> > 
> >  ** We are not done yet, next on the roadmap we have:
> > 
> >  * Streaming: Both of the existing runners lack support for streaming
> >  execution, and currently there is work going on for adding streaming
> >  support to DirectRunner [3].
> >  * Documentation: Filling the rest of the Beam documentations with
> >  Python
> >  SDK specific information and examples.
> >  * SDK consistency: Making Python SDK consistent with the Java SDK.
> We
> >  have
> >  come a long way on this and have only a few items left [4].
> >  * Beamifying: We have been working on removing Dataflow-specific
> >  references
> >  both from the documentation and from the code. There is some work
> > left,
> >  and
> >  we are currently working on those as well [5].
> > 
> >  ** Steps and implications of merging to master:
> > 
> >  * Master branch is merged to python-sdk branch at regular intervals
> > and
> >  the
> >  last merge was on 12/22. All the past merges were uneventful because
> >  there
> >  is a minimal overlap in modified files between branches. Integrating
> >  python-sdk to master will similarly touch a small number of existing
> >  files.
> > 
> > >>>

Re: On my activity at the project

[Frances Perry]

Thanks for all your contributions so far. I hope you enjoy recharging in
your time off and look forward to your return.

I've temporarily updated Jira to put Aljoscha as lead of the Flink
component. But really he or Stephan should opt themselves in to this
responsibility ;-)

On Sat, Jan 14, 2017 at 12:04 AM, Maximilian Michels  wrote:

> Dear Beamers,
>
> Thank you for the past year where we built this amazing community! It's
> been exciting times.
>
> For the beginning of this year, I decided to take some time off. I'd love
> to stay with the project and I think I'm going to be committing more in the
> future. For the meantime, I'd like to pass on the component lead of the
> Flink Runner to either Aljoscha or Stephan who are the most experienced
> Flink committers of the Beam community.
>
> Please feel free to reach out to me in case anything pops up. It's great
> to see Beam as an established top level project. Everyone at the Beam
> community can be really proud!
>
> Best,
> Max
>

Re: Graduation!

[Frances Perry]

I've been part of the evolution of this programming model for a long time
-- from the original FlumeJava prototype, to a key component of Google's
internal infrastructure, to a public cloud product, and now a full-fledged
Apache project. This has been the most exciting, humbling, and challenging
year of that journey.

I'm so proud to be part of this community. Together, we can take this
technology places I never even dreamed of.

On Tue, Jan 10, 2017 at 9:17 AM, Jason Kuster <
jasonkus...@google.com.invalid> wrote:

> Woo h[1]! Congrats to everyone - this is fantastic.
>
>
> [1]
> http://68.media.tumblr.com/3b22c7b2799f4adadbe377af7e8e74
> 43/tumblr_npf7bzwSvA1s2wio8o1_500.gif
>
> On Tue, Jan 10, 2017 at 8:54 AM, Sandeep Deshmukh  >
> wrote:
>
> > Congratulations to all!!!
> >
> > On 10-Jan-2017 9:49 PM, "Amit Sela"  wrote:
> >
> > > Congratulations and thanks to everyone involved!
> > >
> > > On Tue, Jan 10, 2017, 18:16 Aljoscha Krettek 
> > wrote:
> > >
> > > > Sweet! Congratulations to everyone. :-)
> > > >
> > > > On Tue, 10 Jan 2017 at 17:01 Mukul Jain  wrote:
> > > >
> > > > > Congrats to everyone  on this good news !!
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Jan 10, 2017, at 7:59 AM, Ismaël Mejía 
> > wrote:
> > > > > >
> > > > > > Congratulations everyone, this is great news, graduation will
> give
> > > new
> > > > > > users confidence about the project and its community.
> > > > > > Awesome !
> > > > > >
> > > > > > Ismaël
> > > > > >
> > > > > > ps. @Michal It is only one unified API for both bounded and
> > unbounded
> > > > > data,
> > > > > > the behavior changes depending on the data sources.
> > > > > >
> > > > > >> On Tue, Jan 10, 2017 at 4:33 PM, Thomas Weise 
> > > wrote:
> > > > > >>
> > > > > >> Congratulations everyone!
> > > > > >>
> > > > > >> Thomas
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Jan 10, 2017 at 6:14 AM, Bobby Evans
> > > > > 
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Yes great news, Congradulations
> > > > > >>>
> > > > > >>> - Bobby
> > > > > >>>
> > > > > >>> On Tuesday, January 10, 2017, 8:11:43 AM CST, Jungtaek Lim <
> > > > > >>> kabh...@gmail.com> wrote:Congrats all!
> > > > > >>>
> > > > > >>> - Jungtaek Lim (HeartSaVioR)
> > > > > >>>
> > > > > >>> 2017년 1월 10일 (화) 오후 10:53, Andrew Hoblitzell <
> > > > > ahoblitz...@salesforce.com
> > > > >  님이
> > > > > >>> 작성:
> > > > > >>>
> > > > >  Good work all!
> > > > > 
> > > > >  On Tue, Jan 10, 2017 at 8:50 AM, Jesse Anderson <
> > > > > je...@smokinghand.com
> > > > > >>>
> > > > >  wrote:
> > > > > 
> > > > > > Excellent!
> > > > > >
> > > > > >> On Tue, Jan 10, 2017, 7:12 AM Jacky Li 
> > > > wrote:
> > > > > >>
> > > > > >> Great work! Congratulations!
> > > > > >>
> > > > > >> Regards,
> > > > > >> Jacky
> > > > > >>
> > > > > >>> 在 2017年1月10日，下午7:11，Sergio Fernández 
> 写道：
> > > > > >>>
> > > > > >>> Congrats, guys!
> > > > > >>>
> > > > > >>> On Tue, Jan 10, 2017 at 12:07 PM, Davor Bonaci <
> > > da...@apache.org
> > > > > >>>
> > > > > > wrote:
> > > > > >>>
> > > > >  The ASF has publicly announced our graduation!
> > > > > 
> > > > > 
> > > > >  https://blogs.apache.org/foundation/entry/the-apache-
> > > > >  software-foundation-announces
> > > > > 
> > > > >    https://beam.apache.org/blog/
> > 2017/01/10/beam-graduates.html
> > > > > 
> > > > >  Graduation is a recognition of the community that we have
> > > built
> > > > > >> together. I
> > > > >  am humbled to be part of this group and this project, and
> so
> > > > > >>> excited
> > > > > > for
> > > > >  what we can accomplish together going forward.
> > > > > 
> > > > >  Davor
> > > > > 
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> Sergio Fernández
> > > > > >>> Partner Technology Manager
> > > > > >>> Redlink GmbH
> > > > > >>> m: +43 6602747925 <+43%20660%202747925>
> > <+43%20660%202747925>
> > > > > >>> e: sergio.fernan...@redlink.co
> > > > > >>> w: http://redlink.co
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > > 
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
>
>
> --
> ---
> Jason Kuster
> Apache Beam (Incubating) / Google Cloud Dataflow
>