Re: An update on Eugene

[Ekrem Aksoy]

Thank you for all of your contribution. Good luck with your new venture.

On Mon, Jul 16, 2018 at 10:17 PM Eugene Kirpichov 
wrote:

> Hi beamers,
>
> After 5.5 years working on data processing systems at Google, several of
> these years working on Dataflow and Beam, I am moving on to do something
> new (also at Google) in the area of programming models for machine
> learning. Anybody who worked with me closely knows how much I love building
> programming models, so I could not pass up on the opportunity to build a
> new one - I expect to have a lot of fun there!
>
> On the new team we very much plan to make things open-source when the time
> is right, and make use of Beam, just as TensorFlow does - so I will stay in
> touch with the community, and I expect that we will still work together on
> some things. However, Beam will no longer be the main focus of my work.
>
> I've made the decision a couple months ago and have spent the time since
> then getting things into a good state and handing over the community
> efforts in which I have played a particularly active role - they are in
> very capable hands:
> - Robert Bradshaw and Ankur Goenka on Google side are taking charge of
> Portable Runners (e.g. the Portable Flink runner).
> - Luke Cwik will be in charge of the future of Splittable DoFn. Ismael
> Mejia has also been involved in the effort and actively helping, and I
> believe he continues to do so.
> - The Beam IO ecosystem in general is in very good shape (perhaps the best
> in the industry) and does not need a lot of constant direction; and it has
> a great community (thanks JB, Ismael, Etienne and many others!) - however,
> on Google side, Chamikara Jayalath will take it over.
>
> It was a great pleasure working with you all. My last day formally on Beam
> will be this coming Friday, then I'll take a couple weeks of vacation and
> jump right in on the new team.
>
> Of course, if my involvement in something is necessary, I'm still
> available on all the same channels as always (email, Slack, Hangouts) -
> but, in general, please contact the folks mentioned above instead of me
> about the respective matters from now on.
>
> Thanks!
>

Re: [ANNOUCEMENT] New Foundation members!

[Ekrem Aksoy]

Congrats!

On Sat, Mar 31, 2018 at 2:08 AM, Davor Bonaci  wrote:

> Now that this is public... please join me in welcoming three newly elected
> members of the Apache Software Foundation with ties to this community, who
> were elected during the most recent Members' Meeting.
>
> * Ismaël Mejía (Beam PMC)
>
> * Josh Wills (Crunch Chair; Beam, DataFu PMC)
>
> * Holden Karau (Spark, SystemML PMC; Mahout, Subversion committer; Beam
> contributor)
>
> These individuals demonstrated merit in Foundation's growth, evolution,
> and progress. They were recognized, nominated, and elected by existing
> membership for their significant impact to the Foundation as a whole, such
> as the roots of project-related and cross-project activities.
>
> As members, they now become legal owners and shareholders of the
> Foundation. They can vote for the Board, incubate new projects, nominate
> new members, participate in any PMC-private discussions, and contribute to
> any project.
>
> (For the Beam community, this election nearly doubles the number of
> Foundation members. The new members are joining Jean-Baptiste Onofré,
> Stephan Ewen, Romain Manni-Bucau and myself in this role.)
>
> I'm happy to be able to call all three of you my fellow members.
> Congratulations!
>
> Davor
>

Re: IO ITs: Hosting Docker images

[Ekrem Aksoy]

Hi Stephen,

Can we piggyback on current Apache Docker Hub account? I think images can
be hold there, too.

-E

On Mon, Apr 10, 2017 at 5:22 PM, Stephen Sisk 
wrote:

> for 4 - there's a number of logistics involved. How do you propose handling
> cost, potential DOS, etc? People in different timezones would need to be
> oncall for it since it impacts people's ability to dev work (or they need
> to be okay if it goes out.) Can you give some reasons why you think it's
> better than the other options? I put it on the list, but I'm strongly not a
> fan.
>
> S
>
> On Sat, Apr 8, 2017 at 5:31 AM Ted Yu  wrote:
>
> > +1
> >
> > > On Apr 7, 2017, at 10:46 PM, Jean-Baptiste Onofré 
> > wrote:
> > >
> > > Hi Stephen,
> > >
> > > I think we should go to 1 and 4:
> > >
> > > 1. Try to use existing images providing what we need. If we don't find
> > existing image, we can always ask and help other community to provide so.
> > > 4. If we don't find a suitable image, and waiting for this image, we
> can
> > store the image in our own "IT dockerhub".
> > >
> > > Regards
> > > JB
> > >
> > >> On 04/08/2017 01:03 AM, Stephen Sisk wrote:
> > >> Wanted to see if anyone else had opinions on this/provide a quick
> > update.
> > >>
> > >> I think for both elasticsearch and HIFIO that we can find existing,
> > >> supported images that could serve those purposes - HIFIO is looking
> like
> > >> it'll able to do so for cassandra, which was proving tricky.
> > >>
> > >> So to summarize my current proposed solutions: (ordered by my
> > preference)
> > >> 1. (new) Strongly urge people to find existing docker images that meet
> > our
> > >> image criteria - regularly updated/security checked
> > >> 2. Start using helm
> > >> 3. Push our docker images to docker hub
> > >> 4. Host our own public container registry
> > >>
> > >> S
> > >>
> > >>> On Tue, Apr 4, 2017 at 10:16 AM Stephen Sisk 
> wrote:
> > >>>
> > >>> I'd like to hear what direction folks want to go in, and from there
> > look
> > >>> at the options. I think for some of these options (like running our
> own
> > >>> public registry), they may be able to and it's something we should
> > look at,
> > >>> but I don't assume they have time to work on this type of issue.
> > >>>
> > >>> S
> > >>>
> > >>> On Tue, Apr 4, 2017 at 10:00 AM Lukasz Cwik  >
> > >>> wrote:
> > >>>
> > >>> Is this something that Apache infra could help us with?
> > >>>
> > >>> On Mon, Apr 3, 2017 at 7:22 PM, Stephen Sisk  >
> > >>> wrote:
> > >>>
> >  Summary:
> > 
> >  For IO ITs that use data stores that need custom docker images in
> > order
> > >>> to
> >  run, we can't currently use them in a kubernetes cluster (which is
> > where
> > >>> we
> >  host our data stores.) I have a couple options for how to solve this
> > and
> > >>> am
> >  looking for feedback from folks involved in creating IO ITs/opinions
> > on
> >  kubernetes.
> > 
> > 
> >  Details:
> > 
> >  We've discussed in the past that we'll want to allow developers to
> > submit
> >  just a dockerfile, and then we'll use that when creating the data
> > store
> > >>> on
> >  kubernetes. This is the case for ElasticsearchIO and I assume more
> > data
> >  stores in the future will want to do this. It's also looking like
> > it'll
> > >>> be
> >  necessary to use custom docker images for the HadoopInputFormatIO's
> >  cassandra ITs - to run a cassandra cluster, there doesn't seem to
> be a
> > >>> good
> >  image you can use out of the box.
> > 
> >  In either case, in order to retrieve a docker image, kubernetes
> needs
> > a
> >  container registry - it will read the docker images from there. A
> > simple
> >  private container registry doesn't work because kubernetes config
> > files
> > >>> are
> >  static - this means that if local devs try to use the kubernetes
> > files,
> >  they point at the private container registry and they wouldn't be
> > able to
> >  retrieve the images since they don't have access. They'd have to
> > manually
> >  edit the files, which in theory is an option, but I don't consider
> > that
> > >>> to
> >  be acceptable since it feels pretty unfriendly (it is simple, so if
> we
> >  really don't like the below options we can revisit it.)
> > 
> >  Quick summary of the options
> > 
> >  ===
> > 
> >  We can:
> > 
> >  * Start using something like k8 helm - this adds more dependencies,
> > adds
> > >>> a
> >  small amount of complexity (this is my recommendation, but only by a
> >  little)
> > 
> >  * Start pushing images to docker hub - this means they'll be
> publicly
> >  visible and raises the bar for maintenance of those images
> > 
> >  * Host our own public container registry - this means 

Re: Interest in a (virtual) contributor meeting?

[Ekrem Aksoy]

+1
22 Şub 2017 Çar, saat 06:19 tarihinde Davor Bonaci  şunu
yazdı:

> In the early days of the project, we have held a few meetings for the
> initial community to get to know each other. Since then, the community has
> grown a huge amount, but we haven't organized any get-togethers.
>
> I wanted to gauge interest in a potential video conference call in the near
> future. No specific agenda -- simply a chance for everyone to meet others
> and see the faces of people we share a common passion with. Of course, an
> open discussion on any topic of interest to the contributor community is
> welcome. This would be strictly informal -- any decisions are reserved for
> the mailing list discussions.
>
> If you'd be interested in attending, please reply back. If there's
> sufficient interest, I'd be happy to try to organize something in the near
> future.
>
> Thanks!
>
> Davor
>

Re: Beam on Kubernetes

[Ekrem Aksoy]

Hey @Nitin,

We can talk on Slack (I don't know how to get into Slack channel, btw).
I've local vagrant k8s (for different dev purpose, but I can fire a new
cluster based on that). I'm using AWS stack (ECR/ECS for containers and
CloudFormation things)

Best,

Ekrem

On Tue, Feb 21, 2017 at 11:39 AM, Nitin Lamba <nla...@apache.org> wrote:

> @Ekrem
> Sure! Would be great to work together on this.
>
> I'm assuming you already have the basic docker setup on your dev machine.
> If you're new to kubernetes, the first step is to get a local k8s setup
> going - single node (minikube) or multi-node (vagrant). Alternatively, you
> can setup a container environment on GCP or AWS. It'll help packaging and
> test-driving the images/ pods as they're built.
>
> Let me know if you'd like to chat real time (on Slack) about it.
>
> Thanks,
> Nitin
> On Mon, Feb 20, 2017 at 9:41 PM Ekrem Aksoy <ekremak...@gmail.com> wrote:
>
> > I'm also interested in this effort. How can I contribute?
> >
> >
> > Best,
> >
> > Ekrem
> >
> > On Tue, Feb 21, 2017 at 4:12 AM, Nitin Lamba <nitin.la...@gmail.com>
> > wrote:
> >
> > > Thanks JB, Davor!
> > >
> > > Yes, the step-by-step Spark/Mesos guide will certainly help.
> > >
> > > I'll start with packaging java runner/ examples first and see how far I
> > > get. Will create a JIRA and should have an update in the next week or
> so.
> > >
> > > Best,
> > > Nitin
> > >
> > > On Mon, Feb 20, 2017 at 2:05 PM, Davor Bonaci <da...@apache.org>
> wrote:
> > >
> > > > I think these are great ideas for simplifying the getting started
> > > > experience across runners -- we'd love a contribution in this space!
> > > >
> > > > On Mon, Feb 20, 2017 at 12:46 AM, Jean-Baptiste Onofré <
> > j...@nanthrax.net>
> > > > wrote:
> > > >
> > > > > Hi Nitin,
> > > > >
> > > > > It sounds like a good candidate for blog or documentation, or even
> an
> > > > > example.
> > > > >
> > > > > I have a step-by-step example of a pipeline running on Spark with
> > Mesos
> > > > > (not yet Kubernetes) if you are interested.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > >
> > > > > On 02/20/2017 08:47 AM, Nitin Lamba wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> Trying to restart this thread from last November [1]. Packaging an
> > > > >> end-2-end Beam example for k8s environment, similar to the one
> from
> > > the
> > > > >> TensorFlow team [2], may be interesting to look at. The logical
> > > > >> progression
> > > > >> is:
> > > > >>
> > > > >> - Start with an example using the local (java) runner
> > > > >> - Build the next one for Spark; k8s repo already has Spark v1.5.x
> as
> > > an
> > > > >> example [3] that can be updated/ modified
> > > > >> - Other runners to follow using Spark as a template
> > > > >>
> > > > >> Let me know if there is interest in pursuing/ collaborating on
> this.
> > > > >>
> > > > >> Thanks,
> > > > >> Nitin
> > > > >>
> > > > >> [1]
> > > > >> https://www.mail-archive.com/user@beam.incubator.apache.
> > > > org/msg00881.html
> > > > >> [2] https://tensorflow.github.io/serving/serving_inception
> > > > >> [3] https://github.com/kubernetes/kubernetes/tree/master/
> > > examples/spark
> > > > >>
> > > > >>
> > > > > --
> > > > > Jean-Baptiste Onofré
> > > > > jbono...@apache.org
> > > > > http://blog.nanthrax.net
> > > > > Talend - http://www.talend.com
> > > > >
> > > >
> > >
> >
>

Re: Beam on Kubernetes

[Ekrem Aksoy]

I'm also interested in this effort. How can I contribute?


Best,

Ekrem

On Tue, Feb 21, 2017 at 4:12 AM, Nitin Lamba  wrote:

> Thanks JB, Davor!
>
> Yes, the step-by-step Spark/Mesos guide will certainly help.
>
> I'll start with packaging java runner/ examples first and see how far I
> get. Will create a JIRA and should have an update in the next week or so.
>
> Best,
> Nitin
>
> On Mon, Feb 20, 2017 at 2:05 PM, Davor Bonaci  wrote:
>
> > I think these are great ideas for simplifying the getting started
> > experience across runners -- we'd love a contribution in this space!
> >
> > On Mon, Feb 20, 2017 at 12:46 AM, Jean-Baptiste Onofré 
> > wrote:
> >
> > > Hi Nitin,
> > >
> > > It sounds like a good candidate for blog or documentation, or even an
> > > example.
> > >
> > > I have a step-by-step example of a pipeline running on Spark with Mesos
> > > (not yet Kubernetes) if you are interested.
> > >
> > > Regards
> > > JB
> > >
> > >
> > > On 02/20/2017 08:47 AM, Nitin Lamba wrote:
> > >
> > >> Hi,
> > >>
> > >> Trying to restart this thread from last November [1]. Packaging an
> > >> end-2-end Beam example for k8s environment, similar to the one from
> the
> > >> TensorFlow team [2], may be interesting to look at. The logical
> > >> progression
> > >> is:
> > >>
> > >> - Start with an example using the local (java) runner
> > >> - Build the next one for Spark; k8s repo already has Spark v1.5.x as
> an
> > >> example [3] that can be updated/ modified
> > >> - Other runners to follow using Spark as a template
> > >>
> > >> Let me know if there is interest in pursuing/ collaborating on this.
> > >>
> > >> Thanks,
> > >> Nitin
> > >>
> > >> [1]
> > >> https://www.mail-archive.com/user@beam.incubator.apache.
> > org/msg00881.html
> > >> [2] https://tensorflow.github.io/serving/serving_inception
> > >> [3] https://github.com/kubernetes/kubernetes/tree/master/
> examples/spark
> > >>
> > >>
> > > --
> > > Jean-Baptiste Onofré
> > > jbono...@apache.org
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
>