Re: Scio Beam Scala API

Aviem, You're right we have a scio/apache-beam branch that works with Beam 0.2.0-incubating and are working on keeping it up with the latest releases. A few ways you can contribute: - It runs with the Dataflow runner but hasn't been tested with

Re: Beam Interview

Not a committer but I shared some thoughts since we (Spotify) are heavy users of Dataflow/Beam and contribute back to the code base. On Thu, Jul 7, 2016 at 8:18 PM Jesse Anderson wrote: > I've been thinking about ways to get more Beam information out there > without too

Re: Scala DSL

Looks like dsls/scio is the winner :) I like it too plus we get to keep the Scio name. This also leaves room for other Scala wrappers of different flavor. Scio is a DSL in the domain of functional style data pipelines. On Mon, Jun 27, 2016 at 3:55 AM Ismaël Mejía wrote: >

Re: Scala DSL

+folks in my team On Thu, Jun 23, 2016 at 5:57 PM Neville Li <neville@gmail.com> wrote: > Hi all, > > I'm the co-author of Scio <https://github.com/spotify/scio> and am in the > progress of moving code to Beam (BEAM-302 > <https://issues.apache.org/jira/bro

Scala DSL

Hi all, I'm the co-author of Scio and am in the progress of moving code to Beam (BEAM-302 ). Just wondering if sdks/scala is the right place for this code or if something like dsls/scio is a better choice? What do

Re: Apache Zeppelin Beam Integration

The RDD API is only tied to the SparkInterpreter. Scio will probably have its own interpreter but with configurable runner so users can leverage Dataflow (very attractive to us) or Flink also. On Tue, May 17, 2016 at 11:12 AM Jean-Baptiste Onofré wrote: > Hi Ismael, > > If

Re: Apache Zeppelin Beam Integration

The biggest appeal of spark in zeppelin is its interactiveness, i.e. the ability to pull data from RDDs to the driver/web UI via actions (take, collect, top). There are no equivalent of actions in Beam/Dataflow, only transformations (apply(transform)). How's that gonna work with spark? In

Re: [PROPOSAL] New sdk languages

rences would > be to > provide docs like they do in the ReactiveX world with base concepts of the > model > and side notes for language specific syntax. > > http://reactivex.io/documentation/operators/flatmap.html > > Cheers, > Ismaël > > > > On Fri, Apr 1, 2016

Re: [PROPOSAL] New sdk languages

; We are working on some refactoring & polishing on our side (Runner API, > > etc). So, one or two months is not a big deal ! > > > > Let me know if I can help in any way. > > > > Thanks, > > Regards > > JB > > > > > > On 03/25/2016 08:

Re: [PROPOSAL] New sdk languages

eam.incubator.apache.org/capability-matrix/ > > And the new additions to the model are openly discussed in the mailing list > and in the technical docs (e.g. lateness): > > https://goo.gl/ps8twC > > -Ismaël > > On Fri, Mar 25, 2016 at 8:36 AM, Neville Li <neville@gmail.com&

Re: [PROPOSAL] New sdk languages

Thanks guys for the interest. I'm really excited about all the feedbacks from the community. A little background: we developed Scio to bring Google Cloud Dataflow closer to the Scalding/Spark ecosystem that our developers are familiar with while bringing some missing pieces to the table (type