Re: NYC ? (or more generally East Coast)

[Suneel Marthi]

I can do talks in either DC or NYC meetups.  I can coordinate with
CapitalOne to see if they would be willing to host the DC meetup.

On Mon, Jan 13, 2020 at 4:02 PM Austin Bennett 
wrote:

> Hi Devs and Users,
>
> We are looking for speakers for future Meetups and Events.  Who is
> building cool things with Beam?  We are looking at hosting a Meetup at
> Spotify in February, and ideally keep some meetups going throughout
> the year.  For this to occur, we need to hear about what people are
> working on!  Even if only a small/lightning talk, etc, do reach out!
> Let's figure something out.
>
> Cheers,
> Austin
>
> P.S.  https://www.meetup.com/New-York-Apache-Beam/
>
> P.S.S.  We also have budding communities in DC and Boston, will
> eventually write in separate threads on those.
>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[Suneel Marthi]

+1

On Mon, Sep 30, 2019 at 10:33 PM Manu Zhang  wrote:

> +1
>
> On Tue, Oct 1, 2019 at 9:44 AM Austin Bennett 
> wrote:
>
>> +1
>>
>> On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Please vote whether to sign a pledge on behalf of Apache Beam to sunset
>>> Beam Python 2 offering (in new releases) in 2020 on
>>> http://python3stament.org as follows:
>>>
>>> [ ] +1: Sign a pledge to discontinue support of Python 2 in Beam in 2020.
>>> [ ] -1: Do not sign a pledge to discontinue support of Python 2 in Beam
>>> in 2020.
>>>
>>> The motivation and details for this vote were discussed in [1, 2].
>>> Please follow up in [2] if you have any questions.
>>>
>>> This is a procedural vote [3] that will follow the majority approval
>>> rules and will be open for at least 72 hours.
>>>
>>> Thanks,
>>> Valentyn
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>>> [2]
>>> https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
>>> [3] https://www.apache.org/foundation/voting.html
>>>
>>>

Re: Beam Summit Europe: speakers and schedule online!

[Suneel Marthi]

Kulturbraurei is on Schönhauser Allee - u have the address wrong on the
event page.

On Thu, May 23, 2019 at 4:58 PM Joana Filipa Bernardo Carrasqueira <
joanafil...@google.com> wrote:

> Hi all!
>
> Looking forward to the conversations about Beam and to meet new people in
> the community!
>
> Please help us spreading the word about the Beam Summit within your
> networks and register for the event here
> 
> .
>
> See you all soon!
> Joana
>
>
> On Thu, May 23, 2019 at 6:24 AM Matthias Baetens <
> baetensmatth...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Happy to share that the speakers  and
>> schedule  are now online on the
>> website.
>>
>> Make sure you register on Eventbrite
>>  if you want to attend and
>> follow out Twitter channel  for
>> announcements regarding the speakers over the next few weeks!
>>
>> Best regards,
>> Matthias
>>
>
>
> --
>
> *Joana Carrasqueira*
>
> Cloud Developer Relations Events Manager
>
> +1 415-602-2507
>
> 1160 N Mathilda Ave, Sunnyvale, CA 94089
>
>
>

Re: Beam Summit at ApacheCon

[Suneel Marthi]

Could u please further quantify the 'morning pacific time' part of it?  Its
just not clear what the deadline is now from that.

On Sat, May 11, 2019 at 12:47 PM Austin Bennett 
wrote:

> Hi All,
>
> Deadline for CfP is the morning of 13 May (this Monday)  Pacific Time, as
> decided by ApacheCon.  Please submit if you have anything.  Also, do write
> if you have questions/concerns, etc.
>
> Cheers,
> Austin
>
>
>
> On Tue, Apr 30, 2019 at 7:59 AM Austin Bennett <
> whatwouldausti...@gmail.com> wrote:
>
>> Hi Users and Devs,
>>
>> The CfP deadline approaches.  Do submit your technical and/or use case
>> talks, etc etc.  Feel free to reach out if you have any questions.
>>
>> Cheers,
>> Austin
>>
>> On Tue, Apr 23, 2019 at 2:49 AM Maximilian Michels 
>> wrote:
>>
>>> Hi Austin,
>>>
>>> Thanks for the heads-up! I just want to highlight that this is a great
>>> chance for Beam. There will be a _dedicated_ Beam track which means that
>>> there is potential for lots of new people to learn about Beam. Of
>>> course, there will also be many people already involved in Beam.
>>>
>>> -Max
>>>
>>> On 23.04.19 02:47, Austin Bennett wrote:
>>> > Beam Summit will be at ApacheCon this year -- please consider
>>> submitting!
>>> >
>>> > Dates for Beam Summit 11 and 12 September 2019.  There are other
>>> tracks
>>> > at ApacheCon during this and on other dates too.
>>> >
>>> > https://www.apachecon.com/acna19/cfp.html
>>> >
>>> >
>>>
>>

Re: Hello from Hannah Jiang

[Suneel Marthi]

Welcome!!

On Fri, Apr 26, 2019 at 12:58 PM Lukasz Cwik  wrote:

> Welcome Hannah.
>
> On Fri, Apr 26, 2019 at 5:00 AM Ismaël Mejía  wrote:
>
>> Welcome to Beam Hannah !
>>
>> On Fri, Apr 26, 2019 at 1:44 PM Maximilian Michels 
>> wrote:
>> >
>> > Awesome. Welcome Hannah!
>> >
>> > Cheers,
>> > Max
>> >
>> > On 26.04.19 05:08, Yifan Zou wrote:
>> > > Welcome！
>> > >
>> > > On Thu, Apr 25, 2019 at 7:34 PM Connell O'Callaghan <
>> conne...@google.com
>> > > > wrote:
>> > >
>> > > Welcome Hannah!!!
>> > >
>> > > On Thu, Apr 25, 2019, 5:42 PM Reza Rokni > > > > wrote:
>> > >
>> > > Welcome!
>> > >
>> > > On Fri, 26 Apr 2019 at 04:36, Hannah Jiang
>> > > mailto:hannahji...@google.com>>
>> wrote:
>> > >
>> > > Thanks Cyrus!
>> > >
>> > > On Thu, Apr 25, 2019 at 1:34 PM Cyrus Maden
>> > > mailto:cma...@google.com>> wrote:
>> > >
>> > > Welcome!!
>> > >
>> > > On Thu, Apr 25, 2019 at 4:30 PM Hannah Jiang
>> > > > hannahji...@google.com>>
>> > > wrote:
>> > >
>> > > Thank you Robin!
>> > >
>> > > On Thu, Apr 25, 2019 at 1:27 PM Robin Qiu
>> > > mailto:robi...@google.com>>
>> wrote:
>> > >
>> > > Welcome Hannah!
>> > >
>> > > On Thu, Apr 25, 2019 at 1:26 PM Hannah Jiang
>> > > > > > > wrote:
>> > >
>> > > Thanks Kenneth!
>> > >
>> > > On Thu, Apr 25, 2019 at 1:24 PM Kenneth
>> > > Knowles > > > > wrote:
>> > >
>> > > Welcome!
>> > >
>> > > On Thu, Apr 25, 2019 at 12:38 PM
>> > > Matthias Baetens
>> > > > > > >
>> wrote:
>> > >
>> > > Welcome to the community!
>> > >
>> > > On Thu, Apr 25, 2019, 18:55
>> Griselda
>> > > Cuevas > > > > wrote:
>> > >
>> > > Welcome Hannah! - Very excited
>> > > to see you in the Beam
>> community :)
>> > >
>> > > On Tue, 23 Apr 2019 at 12:59,
>> > > Hannah Jiang
>> > > > > > > hannahji...@google.com>>
>> > > wrote:
>> > >
>> > > Hi everyone
>> > >
>> > > I joined Google recently
>> and
>> > > would work on Python
>> > > portability part. I am
>> happy
>> > > to be part of the
>> community.
>> > > Looking forward to working
>> > > with all of you together.
>> > >
>> > > I have a minor request,
>> can
>> > > admin please give me
>> access
>> > > to JIRA?
>> > >
>> > > Thanks,
>> > > Hannah
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > This email may be confidential and privileged. If you received
>> > > this communication by mistake, please don't forward it to
>> anyone
>> > > else, please erase all copies and attachments, and please let
>> me
>> > > know that it has gone to the wrong person.
>> > >
>> > > The above terms reflect a potential business arrangement, are
>> > > provided solely as a basis for further discussion, and are not
>> > > intended to be and do not constitute a legally binding
>> > > obligation. No legally binding obligations will be created,
>> > > implied, or inferred until an agreement in final form is
>> > > executed in writing by all parties involved.
>> > >
>>
>

Re: Connected streams with Beam

[Suneel Marthi]

Thanks, CoGroupByKey is what i need.

On Tue, Mar 19, 2019 at 12:24 PM Maximilian Michels  wrote:

> Hi Suneel,
>
> Depending on what exactly you want to to, please have a look at Flatten
> or CoGroupByKey:
>
> https://beam.apache.org/documentation/programming-guide/#flatten
> https://beam.apache.org/documentation/programming-guide/#cogroupbykey
>
> Both are available in Python/Java.
>
> -Max
>
> On 18.03.19 23:37, Suneel Marthi wrote:
> > Could someone point me to how to do connected stream from 2 sources with
> both java and python api?
> >
> > Danke
> >
> > Sent from my iPhone
> >
>

Connected streams with Beam

[Suneel Marthi]

Could someone point me to how to do connected stream from 2 sources with both 
java and python api?

Danke

Sent from my iPhone

Re: Apache Beam Newsletter - February/March 2019

[Suneel Marthi]

I believe there was also a Beam workshop or working session in Warsaw last
week.

On Wed, Mar 6, 2019 at 6:20 PM Austin Bennett 
wrote:

> +1 for archive in our repo.
>
> I do follow the newsletter, but am unlikely to go back and look into the
> past for changes/updates.
>
> Would suggest that things that get missed in one newsletter (a concrete
> example, Suneel's talks not mentioned in the newsletter) would get
> published in the next iteration, rather than editing the past 'published'
> newsletter.  Put another way, save editing the past for corrections (typos,
> things being incorrect).  Else, I imagine that I'm unlikely to catch a
> great announcement that warranted being in the newsletter in the first
> place.  This certainly works better with a regular/frequent release
> cadence, like we arrived at for version releases (then, if something misses
> one cut, it is not too big a deal, as the next release is coming soon).
>
>
>
>
> On Wed, Mar 6, 2019 at 12:50 PM Melissa Pashniak 
> wrote:
>
>>
>> For step #2 (publishing onto the website), I think it would be good to
>> stay consistent with our existing workflows if possible. Rather than using
>> an external tool, what about:
>>
>> After a google doc newsletter draft is ready, convert it into a standard
>> markdown file and put it into our GitHub repo, perhaps in a new newsletter
>> directory in the website community directory [1]. These would be listed for
>> browsing on a Newsletters page as mentioned in step #4. People can then
>> just open a PR to add missing things to the pages later, and the newsletter
>> will be automatically updated on the website through our standard website
>> workflow. It also avoids the potential issue of the source google docs
>> disappearing in the future, as they are stored in a community location.
>>
>> [1] https://github.com/apache/beam/tree/master/website/src/community
>>
>>
>> On Wed, Mar 6, 2019 at 10:36 AM Rose Nguyen  wrote:
>>
>>> I think that would be a great idea to change formats to help with
>>> distribution. I'm open to suggestions! I'm currently using a Google doc to
>>> collect and edit, then copy/paste sending the newsletter out directly, based
>>> on an interpretation of this discussion
>>> <https://lists.apache.org/thread.html/1f638eae43fe8abcb2f8752141c96d3dbdac86a583e0790044eea727@%3Cdev.beam.apache.org%3E>
>>> .
>>>
>>> How about this doc->website->Beam site workflow?:
>>>
>>>1. The same usual newsletter [CALL FOR ITEMS] where you can
>>>contribute to the google doc, with soft deadlines for when I'll publish.
>>>2. I'll publish the doc itself onto a website.
>>>3. The newsletter is mailed out in the same way, but now with a
>>>shareable website link.
>>>4. We'll keep an index of archived newsletter web pages on the Beam
>>>site, under the Community tab.
>>>5. If you want to submit more content after the soft deadline, add
>>>it to the google doc and let me know to republish. I don't want to make 
>>> the
>>>publication changes automatic because that leaves us open to tampering.
>>>
>>>
>>> This process is more laggy, so I'd suggest doing a 2 month vs monthly
>>> newsletter cadence. If we're happy with this idea, I'll send in a website
>>> PR for a new "Newsletter" left nav item under Community.
>>>
>>> Here's an example of a published newsletter: Apache Beam February-March
>>> 2019
>>> <https://gdoc.pub/doc/e/2PACX-1vTQIS4WkxV-HpgX5Lb6q05g4-wuIVcYd82123Mp4Y6q9fMv6Ynwd-l7dI4TrMyCrKilyU-YsoitbnZB>
>>>
>>>
>>>- This link is permanent unless the principal google doc is deleted.
>>>- Changes to the google doc after web publication are not
>>>automatically published on the website to protect the information 
>>> integrity.
>>>- Republishing is quick and easy for me if you let me know you've
>>>added more.
>>>- I'll improve the formatting later if we go with this route.
>>>
>>> Any thoughts?
>>>
>>> On Wed, Mar 6, 2019 at 6:13 AM Thomas Weise  wrote:
>>>
>>>> Similar to blog posts. A link that can be shared would also help to
>>>> distribute over other channels, such as Twitter.
>>>>
>>>>
>>>> On Wed, Mar 6, 2019, 6:06 AM Ismaël Mejía  wrote:
>>>>
>>>>> We should have these newsletters published somewhere with a fixed URL
>>>>> so we can add missing updates, I h

Re: Apache Beam Newsletter - February/March 2019

[Suneel Marthi]

Is this the final draft? - we had 2 beam talks at Big Data Tech Warsaw last
Wednesday - I can send the updates offline.

On Mon, Mar 4, 2019 at 6:16 PM Rose Nguyen  wrote:

>
> [image: Beam.png]
>
> February-March 2019 | Newsletter
>
> What’s been done
>
> --
>
> Apache Beam 2.10.0 released (by: many contributors)
>
>-
>
>Download the release here.
>
>-
>
>See the blog post
> for more
>details.
>
>
> Apache Beam awarded the 2019 Technology of the Year Award!
>
>-
>
>InfoWorld just awarded Beam the 2019 Technology of the Year Award.
>-
>
>See this  article
>
> 
>for more details.
>
>
> Kettle Beam 0.5 released with support for flink (by: Matt Casters)
>
>-
>
>Kettle now supports Apache Flink as well as Cloud Dataflow and Spark.
>-
>
>See Matt’s Blog
>
> 
>for more details.
>
>
>
> What we’re working on...
>
> --
>
> Apache Beam 2.11.0 release (by: many contributors)
>
>
> Hive Metastore Table provider for SQL (by: Anton Kedin)
>
>-
>
>Support for plugging table providers through Beam SQL API to allow
>obtaining table schemas from external sources.
>-
>
>See the PR  for more details.
>
>
> User Defined Coders for the Beam Go SDK (by: Robert Burke)
>
>-
>
>Working on expanding the variety of user defined types that can be a
>member of a PCollection in the Go SDK.
>-
>
>See BEAM-3306  for
>more details.
>
>
> Python 3 (by: Ahmet Altay, Robert Bradshaw, Charles Chen, Mark Liu, Robbe
> Sneyders, Juta Staes, Valentyn Tymofieiev)
>
>-
>
>Beam 2.11.0 is the first release offering partial Python 3 support.
>-
>
>Many thanks to all contributors who helped to reach this milestone.
>-
>
>IO availablility on Python 3 is currently limited and only Python 3.5
>version has been tested extensively.
>-
>
>Stay tuned on BEAM-1251 for more details.
>
>
> Notebooks for quickstarts and custom I/O (by: David Cavazos)
>
>-
>
>Adding IPython notebooks and snippets
>-
>
>See [BEAM-6557]  for more
>details.
>
>
>
>
>  New members
> --
>
> New PMC member!
>
>-
>
>Etienne Chauchot, Nantes, France
>
>
> New Committers!
>
>-
>
>Gleb Kanterov, Stockholm, Sweden
>-
>
>Michael Luckey
>
>
> New Contributors!
>
>-
>
>Kyle Weaver, San Francisco, CA
>-
>
>   Would like to help begin implementing portability support for the
>   Spark runner
>   -
>
>Tanay Tummapalli, Delhi, India
>-
>
>   Would like to contribute to Open Source this summer as part of
>   Google Summer of Code
>   -
>
>Brian Hulette, Seattle, WA
>-
>
>   Contributing to Beam Portability
>   -
>
>Michał Walenia, Warsaw, Poland
>-
>
>   Working on integration and load testing
>   -
>
>Daniel Chen, San Francisco, CA
>-
>
>   Working on Beam Samza runner
>
>
>
>  Talks & meetups
> --
>
>
> Plugin Machine Intelligence and Apache Beam with Pentaho - Feb 7 @ London
>
>-
>
>Watch the How to Run Kettle on Apache Beam video here
>
> .
>
>-
>
>See event details here
>..
>
>
> Beam @Lyft / Streaming, TensorFlow and use-cases - Feb 7 @ San Francisco,
> CA
>
>-
>
>Organized by Thomas Weise and Austin Bennet, with speakers Tyler
>Akidau, Robert Crowe, Thomas Weise and Amar Pai
>-
>
>See event details here
>
>and the slides for these presentation: Overview of Apache Beam and
>TensorFlow Transform (TFX) with Apache Beam
>, Python Streaming Pipelines
>with Beam on Flink , 
> Dynamic
>pricing of Lyft rides using streaming
>
> 
>
> .
>
> Flink meetup - Feb 21@ Seattle, WA
>
>-
>
>Speakers from Alibaba, Google, and Uber gave talks about Apache Flink
>with Hive, Tensorflow, Beam, and AthenaX.
>-
>
>See event details here
> and
>presentations here .
>
>
>
> Beam 

Re: s3 filesystem for Python good for GSoC?

[Suneel Marthi]

Yup, something like this.

import boto3 s3r = boto3.resource(“s3”) data = s3r.Object(bucket=“bucket”,
key=“key”).read()


On Thu, Feb 21, 2019 at 9:50 PM Boyuan Zhang  wrote:

> I believe the Boto3 lib should be helpful with right credential
> configuration when creating a client:
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration
>
> On Thu, Feb 21, 2019 at 6:15 PM Suneel Marthi 
> wrote:
>
>> Couldn't u just use Boto python package for doing that ?
>>
>> I am writing one now to read from S3 via the Python api
>>
>> On Thu, Feb 21, 2019 at 6:19 PM Pablo Estrada  wrote:
>>
>>> Hello all,
>>> I was thinking that a filesystem with support for s3 would be great to
>>> have in the Python SDK. If I am not wrong, it would simply involve
>>> implementing the filesystem classes
>>> <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/filesystem.py>
>>>  with
>>> s3, right?
>>>
>>> I am not familiar enough with s3, nor with filesystems, nor with AWS in
>>> general - but I have some outstanding questions:
>>>
>>>- Does this mean that we probably would need an extra [s3] target
>>>for installing apache_beam, like we do with [gcp]?
>>>   - Not strictly necessary, but probably desirable...
>>>- How do we handle KMS in GCS filesystem?
>>>- Would the filesystem encapsulation make KMS support in an s3
>>>filesystem difficult?
>>>- Or even more... is the KMS support in AWS very different than in
>>>GCP?
>>>   - I'd love comments from anyone informed around this : )
>>>- Is this project of an appropriate size for a GSoC student?
>>>
>>> Thoughts?
>>> Best
>>> -P.
>>>
>>

Re: s3 filesystem for Python good for GSoC?

[Suneel Marthi]

Couldn't u just use Boto python package for doing that ?

I am writing one now to read from S3 via the Python api

On Thu, Feb 21, 2019 at 6:19 PM Pablo Estrada  wrote:

> Hello all,
> I was thinking that a filesystem with support for s3 would be great to
> have in the Python SDK. If I am not wrong, it would simply involve
> implementing the filesystem classes
> 
>  with
> s3, right?
>
> I am not familiar enough with s3, nor with filesystems, nor with AWS in
> general - but I have some outstanding questions:
>
>- Does this mean that we probably would need an extra [s3] target for
>installing apache_beam, like we do with [gcp]?
>   - Not strictly necessary, but probably desirable...
>- How do we handle KMS in GCS filesystem?
>- Would the filesystem encapsulation make KMS support in an s3
>filesystem difficult?
>- Or even more... is the KMS support in AWS very different than in GCP?
>   - I'd love comments from anyone informed around this : )
>- Is this project of an appropriate size for a GSoC student?
>
> Thoughts?
> Best
> -P.
>

Re: [ANNOUNCE] New committer announcement: Gleb Kanterov

[Suneel Marthi]

Congratulations

On Fri, Jan 25, 2019 at 12:04 PM Anton Kedin  wrote:

> Congrats!
>
> On Fri, Jan 25, 2019 at 8:54 AM Ismaël Mejía  wrote:
>
>> Well deserved, congratulations Gleb!
>>
>> On Fri, Jan 25, 2019 at 10:47 AM Etienne Chauchot 
>> wrote:
>> >
>> > Congrats Gleb and welcome onboard !
>> >
>> > Etienne
>> >
>> > Le vendredi 25 janvier 2019 à 10:39 +0100, Alexey Romanenko a écrit :
>> >
>> > Congrats to Gleb and welcome on board!
>> >
>> > On 25 Jan 2019, at 09:22, Tim Robertson 
>> wrote:
>> >
>> > Welcome Gleb and congratulations!
>> >
>> > On Fri, Jan 25, 2019 at 8:06 AM Kenneth Knowles  wrote:
>> >
>> > Hi all,
>> >
>> > Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Gleb Kanterov
>> >
>> > Gleb started contributing to Beam and quickly dove deep, doing some
>> sensitive fixes to schemas, also general build issues, Beam SQL, Avro, and
>> more. In consideration of Gleb's technical and community contributions, the
>> Beam PMC trusts Gleb with the responsibilities of a Beam committer [1].
>> >
>> > Thank you, Gleb, for your contributions.
>> >
>> > Kenn
>> >
>> > [1]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> >
>> >
>>
>

Re: Beam Summits!

[Suneel Marthi]

How about Beam Summit in Berlin on Sep 6 immediately following Flink
Forward Berlin on the previous 2 days.

Same may be for Asia also following Flink Forward Asia where and whenever
it happens.

On Wed, Dec 19, 2018 at 6:06 PM Austin Bennett 
wrote:

> Hi All,
>
> I really enjoyed Beam Summit in London (Thanks Matthias!), and there was
> much enthusiasm for continuations.  We had selected that location in a
> large part due to the growing community there, and we have users in a
> variety of locations.  In our 2019 calendar,
> https://docs.google.com/spreadsheets/d/1CloF63FOKSPM6YIuu8eExjhX6xrIiOp5j4zPbSg3Apo/
> shared in the past weeks, 3 Summits are tentatively slotted for this year.
> Wanting to start running this by the group to get input.
>
> * Beam Summit NA, in San Francisco, approx 3 April 2019 (following Flink
> Forward).  I can organize.
> * Beam Summit Europe, in Stockholm, this was the runner up in voting
> falling behind London.  Or perhaps Berlin?  October-ish 2019
> * Beam Summit Asia, in Tokyo ??
>
> What are general thoughts on locations/dates?
>
> Looking forward to convening in person soon.
>
> Cheers,
> Austin
>

Re: Stand at FOSDEM 2019

[Suneel Marthi]

+1

On Thu, Nov 29, 2018 at 6:14 AM Matthias Baetens 
wrote:

> Hey Max,
>
> Great idea. I'd be very keen to join. I'll look at my calendar over the
> weekend to see if this would work.
> Are you going yourself?
>
> Cheers,
> Matthias
>
> On Thu, 29 Nov 2018 at 11:06 Maximilian Michels  wrote:
>
>> Hi,
>>
>> For everyone who might be attending FOSDEM19: What do you think about
>> taking a slot for Beam at the Apache stand?
>>
>> A slot is 2-3 hours. It is a great way to spread the word about Beam. We
>> wouldn't have to prepare much, just bring some merch.
>>
>> There is still plenty of space:
>> https://cwiki.apache.org/confluence/display/COMDEV/FOSDEM+2019
>>
>> Cheers,
>> Max
>>
>> PS: FOSDEM is an open-source conference in Brussels, Feb 2-3, 2019
>>
> --
>
>

Re: E-mail Organization

[Suneel Marthi]

Kafka uses KIPs
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

Flink uses FLIPs
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

So Beam - BIPs 

On Mon, Nov 19, 2018 at 10:48 PM Lukasz Cwik  wrote:

> dev@beam.apache.org gets a lot of e-mail. I was wondering how other
> Apache projects help their contributors focus on design/project discussions
> (such as SQL, SplittableDoFn, Portability, Samza, Flink, Testing, ...)?
>
> I'm looking for a solution that allows people to tag a discussion with
> multiple topics, and that tags can be added after the e-mail has been sent
> as the discussion may cross multiple topics such as testing and SQL.
>
> I was initially thinking that we could embed tags like "topic:sql" in the
> message body and if something was part of multiple tags it would be
> "topic:sql topic:testing" to make it easy
>
> What do you think?
>

Re: [ANNOUNCE] New committer announcement, Euphoria edition

[Suneel Marthi]

Congrats!

On Thu, Nov 1, 2018 at 12:49 PM Ahmet Altay  wrote:

> Congratulations!
>
> On Thu, Nov 1, 2018 at 9:36 AM, Tim  wrote:
>
>> Congratulations and welcome!
>>
>> Tim
>>
>> On 1 Nov 2018, at 17:06, Matthias Baetens 
>> wrote:
>>
>> Congrats David!!!
>>
>> On Thu, Nov 1, 2018, 16:04 Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new committer
>>> :
>>>
>>>  - David Morávek: of, but not limited to, the new Euphoria API
>>>
>>> Through his work with us merging the Euphoria API, community outreach,
>>> and other contributions to Beam, the PMC trusts David with the
>>> responsibilities of a Beam committer [1].
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/become-a-committer/
>>> #an-apache-beam-committer
>>>
>>> --
>>
>>
>>
>

Re: [DISCUSS] - Separate JIRA notifications to a new mailing list

[Suneel Marthi]

+1

Sent from my iPhone

> On Oct 11, 2018, at 8:03 PM, Łukasz Gajowy  wrote:
> 
> This is a good idea. +1
> 
> Łukasz
> 
> 
> czw., 11 paź 2018, 18:01 użytkownik Udi Meiri  napisał:
>> +1 to split JIRA notifications
>> 
>>> On Thu, Oct 11, 2018 at 9:13 AM Kenneth Knowles  wrote:
>>> 
 On Thu, Oct 11, 2018 at 9:10 AM Mikhail Gryzykhin 
  wrote:
 +1.
 Should we separate Jenkins notifications as well?
>>> 
>>> I'm worried this question will get buried in the thread. Would you mind 
>>> separating it into another thread if you would like to discuss?
>>> 
>>> Kenn
>>>  
>>>  
> On Thu, Oct 11, 2018, 08:59 Scott Wegner  +1, commits@ is too noisy to be useful currently.
> 
>> On Thu, Oct 11, 2018 at 8:04 AM Maximilian Michels  
>> wrote:
>> +1
>> 
>> I guess most people have already filters in place to separate commits 
>> and JIRA issues. JIRA really has nothing to do in the commits list.
>> 
>> On 11.10.18 15:53, Kenneth Knowles wrote:
>> > +1
>> > 
>> > I've suggested the same. Canonical.
>> > 
>> > On Thu, Oct 11, 2018, 06:19 Thomas Weise > > > wrote:
>> > 
>> > +1
>> > 
>> > 
>> > On Thu, Oct 11, 2018 at 6:18 AM Etienne Chauchot
>> > mailto:echauc...@apache.org>> wrote:
>> > 
>> > +1 for me also, my gmail filters list is kind of overflowed :)
>> > 
>> > Etienne
>> > 
>> > Le jeudi 11 octobre 2018 à 14:44 +0200, Robert Bradshaw a 
>> > écrit :
>> >> Huge +1 from me too.
>> >> On Thu, Oct 11, 2018 at 2:42 PM Jean-Baptiste Onofré 
>> >> mailto:j...@nanthrax.net>> wrote:
>> >>
>> >> +1
>> >>
>> >> We are doing the same in Karaf as well.
>> >>
>> >> Regards
>> >> JB
>> >>
>> >> On 11/10/2018 14:35, Colm O hEigeartaigh wrote:
>> >> Hi all,
>> >>
>> >> Apologies in advance if this has already been discussed (and 
>> >> rejected).
>> >> I was wondering if it would be a good idea to create a new 
>> >> mailing list
>> >> and divert the JIRA notifications to it? Currently
>> >> "comm...@beam.apache.org   
>> >> >" 
>> >> receives both
>> >> the git and JIRA notifications, and has a huge volume of 
>> >> traffic as a
>> >> result.
>> >>
>> >> Separating JIRA notifications from commit messages would 
>> >> allow users to
>> >> subscribe to whichever are of interest without having to 
>> >> write a mail
>> >> filter if e.g. they are not interested in JIRA notifications. 
>> >> It also
>> >> seems a bit unintuitive to me to expect JIRA notifications to 
>> >> go to an
>> >> email list called "commits".
>> >>
>> >> As a reference point - Apache CXF maintains a "commits" list 
>> >> for git
>> >> notifications and "issues" for JIRA notifications:
>> >>
>> >> http://cxf.apache.org/mailing-lists.html
>> >>
>> >> Thanks!
>> >>
>> >> Colm.
>> >>
>> >> --
>> >> Colm O hEigeartaigh
>> >>
>> >> Talend Community Coder
>> >> http://coders.talend.com
>> >>
>> >> --
>> >> Jean-Baptiste Onofré
>> >> jbono...@apache.org 
>> >> http://blog.nanthrax.net
>> >> Talend -http://www.talend.com
>> > 
> 
> 
> -- 
> 
> 
> 
> 
> Got feedback? tinyurl.com/swegner-feedback

Re: Agenda for the Beam Summit London 2018

[Suneel Marthi]

Looking forward to the conference.

On Sat, Sep 29, 2018 at 10:07 AM Matthias Baetens 
wrote:

> Looking forward to have you all.
>
> @Andrew: we are trying to cover as much sessions with recording as
> possible. Stay tuned and check the Beam YouTube channel after the event!
>
> On Fri, 28 Sep 2018 at 19:51 Danny Angus  wrote:
>
>>
>> How exciting, can't wait to join you guys on Monday!
>> :-)
>> D.
>>
>> On 2018/09/27 22:03:16, Griselda Cuevas  wrote:
>> > Hi Beam Community,
>> >
>> > We have finalized the agenda for the Beam Summit London 2018, it's here:
>> >
>> https://www.linkedin.com/feed/update/urn:li:activity:6450125487321735168/
>> >
>> >
>> > We had a great amount of talk proposals, thank you so much to everyone
>> who
>> > submitted one! We also sold out the event, so we're very excited to see
>> the
>> > community growing.
>> >
>> >
>> > See you around,
>> >
>> > Gris on behalf of the Organizing Committee
>> >
>>
> --
>
>

Re: Donating the Dataflow Worker code to Apache Beam

[Suneel Marthi]

+1 (non-googler)

This is a great  move

Sent from my iPhone

> On Sep 13, 2018, at 2:25 PM, Tim Robertson  wrote:
> 
> +1 (non googler)
> It sounds pragmatic, helps with transparency should issues arise and enables 
> more people to fix. 
>  
> 
>> On Thu, Sep 13, 2018 at 8:15 PM Dan Halperin  wrote:
>> From my perspective as a (non-Google) community member, huge +1.
>> 
>> I don't see anything bad for the community about open sourcing more of the 
>> probably-most-used runner. While the DirectRunner is probably still the most 
>> referential implementation of Beam, can't hurt to see more working code. 
>> Other runners or runner implementors can refer to this code if they want, 
>> and ignore it if they don't.
>> 
>> In terms of having more code and tests to support, well, that's par for the 
>> course. Will this change make the things that need to be done to support 
>> them more obvious? (E.g., "this PR is blocked because someone at Google on 
>> Dataflow team has to fix something" vs "this PR is blocked because the 
>> Apache Beam code in foo/bar/baz is failing, and anyone who can see the code 
>> can fix it"). The latter seems like a clear win for the community.
>> 
>> (As long as the code donation is handled properly, but that's completely 
>> orthogonal and I have no reason to think it wouldn't be.)
>> 
>> Thanks,
>> Dan
>> 
>>> On Thu, Sep 13, 2018 at 11:06 AM Lukasz Cwik  wrote:
>>> Yes, I'm specifically asking the community for opinions as to whether it 
>>> should be accepted or not.
>>> 
 On Thu, Sep 13, 2018 at 10:51 AM Raghu Angadi  wrote:
 This is terrific! 
 
 Is thread asking for opinions from the community about if it should be 
 accepted? Assuming Google side decision is made to contribute, big +1 from 
 me to include it next to other runners. 
 
> On Thu, Sep 13, 2018 at 10:38 AM Lukasz Cwik  wrote:
> At Google we have been importing the Apache Beam code base and 
> integrating it with the Google portion of the codebase that supports the 
> Dataflow worker. This process is painful as we regularly are making 
> breaking API changes to support libraries related to running portable 
> pipelines (and sometimes in other places as well). This has made it 
> sometimes difficult for PR changes to make changes without either 
> breaking something for Google or waiting for a Googler to make the change 
> internally (e.g. dependency updates).
> 
> This code is very similar to the other integrations that exist for 
> runners such as Flink/Spark/Apex/Samza. It is an adaption layer that sits 
> on top of an execution engine. There is no super secret awesome stuff as 
> this code was already publicly visible in the past when it was part of 
> the Google Cloud Dataflow github repo[1].
> 
> Process wise the code will need to get approval from Google to be donated 
> and for it to go through the code donation process but before we attempt 
> to do that, I was wondering whether the community would object to adding 
> this code to the master branch?
> 
> The up side is that people can make breaking changes and fix it for all 
> runners. It will also help Googlers contribute more to the portability 
> story as it will remove the burden of doing the code import (wasted time) 
> and it will allow people to develop in master (can have the whole project 
> loaded in a single IDE).
> 
> The downsides are that this will represent more code and unit tests to 
> support.
> 
> 1: 
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/hotfix_v1.2/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #2

[Suneel Marthi]

+1 non-binding

1. verified Sigs and Hashes of artifacts
2. tested with my sample applications with local Runner

On Sun, Aug 5, 2018 at 12:47 AM, Jean-Baptiste Onofré 
wrote:

> +1 (binding)
>
> Tested with beam-samples, checksum and sig verified.
>
> Regards
> JB
>
> On 04/08/2018 01:27, Pablo Estrada wrote:
> > Hello everyone!
> >
> > Extra, extra! The Apache Beam 2.6.0 release candidate #2 is out.
> >
> > Please review and vote on the release candidate #2 for the version
> > 2.6.0, as follows:
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staged set of artifacts is available for your review, which
> > includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> >  [2], which is signed with the key with
> > fingerprint 2F1FEDCDF6DD7990422F482F65224E0292DD8A51 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.6.0-RC2" [5],
> > * website pull request listing the release and publishing the API
> > reference manual [6]. This did not change from the previous RC.
> > * Python artifacts are deployed along with the source release to
> > the dist.apache.org  [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Regards
> > -Pablo.
> >
> > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527=12343392
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.6.0/
> > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> > [4] https://repository.apache.org/content/repositories/
> orgapachebeam-1045/
> > [5] https://github.com/apache/beam/tree/v2.6.0-RC2
> > [6] https://github.com/apache/beam-site/pull/518
> > --
> > Got feedback? go/pabloem-feedback
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

[Suneel Marthi]

+1 non-binding

1. tested with beam samples
2. verified sigs and hashes of artifacts


On Fri, Aug 3, 2018 at 12:43 AM, Jean-Baptiste Onofré 
wrote:

> +1 (binding)
>
> Tested with beam-samples.
>
> I didn't have time to include three Jira, but 2.7.0 should be in vote in
> soon ;)
>
> Regards
> JB
>
> On 01/08/2018 01:50, Pablo Estrada wrote:
> > Hello everyone!
> >
> > I have been able to prepare a release candidate for Beam 2.6.0. : D
> >
> > Please review and vote on the release candidate #1 for the version
> > 2.6.0, as follows:
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staged set of artifacts is available for your review, which
> > includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> >  [2], which is signed with the key with
> > fingerprint 2F1FEDCDF6DD7990422F482F65224E0292DD8A51 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.6.0-RC1" [5],
> > * website pull request listing the release and publishing the API
> > reference manual [6].
> > * Python artifacts are deployed along with the source release to the
> > dist.apache.org  [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Regards
> > -Pablo.
> >
> > [1]
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527=12343392
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.6.0/
> > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> > [4] https://repository.apache.org/content/repositories/
> orgapachebeam-1044/
> > [5] https://github.com/apache/beam/tree/v2.6.0-RC1
> > [6] https://github.com/apache/beam-site/pull/518
> >
> > --
> > Got feedback? go/pabloem-feedback
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Live coding & reviewing adventures

[Suneel Marthi]

Could u publish the python transitive deps some place that have the
Beam-Flink runner working ?

On Tue, Jul 17, 2018 at 5:26 PM, Holden Karau  wrote:

> And I've got an hour to kill @ SFO today so at some of the suggestions
> from folks I'm going to do a more user focused one trying getting the TFT
> demo to work with the portable flink runner (hopefully) -
> https://www.youtube.com/watch?v=wL9mvQeN36E
>
> On Fri, Jul 13, 2018 at 11:54 AM, Holden Karau 
> wrote:
>
>> Hi folks! I've been doing some live coding in my other projects and I
>> figured I'd do some with Apache Beam as well.
>>
>> Today @ 3pm pacific I'm going be doing some impromptu exploration better
>> review tooling possibilities (looking at forking spark-pr-dashboard for
>> other projects like beam and setting up mentionbot to work with ASF infra)
>> - https://www.youtube.com/watch?v=ff8_jbzC8JI
>>
>> Next week (Thursday the 19th at 2pm pacific) I'm going to be working on
>> trying to get easier dependency management for the Python portable runner
>> in place - https://www.youtube.com/watch?v=Sv0XhS2pYqA
>>
>> If your interested in seeing more of the development process I hope you
>> will join me :)
>>
>> P.S.
>>
>> You can also follow on twitch which does a better job of notifications
>> https://www.twitch.tv/holdenkarau
>>
>> Also one of the other thing I do is "live reviews" of PRs but they are
>> generally opt-in and I don't have enough opt-ins from the Beam community to
>> do live reviews in Beam, if you work on Beam and would be OK with me doing
>> a live streamed review of your PRs let me know (if your curious to what
>> they look like you can see some of them here in Spark land
>> 
>> ).
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>>
>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
>