Re: Review on Jira for 0.4.0-incubating

Sounds great, JB! The major blocker in my opinion is to finish the polishing pass on the quickstarts and example archetypes, so that users will have a great experience trying out 0.4.0-incubating. I know we've made some significant progress there in the last few weeks, but I don't think we've quit

Re: Unable to clone beam site

Make sure to checkout the Website section of the contribution guide that contains all the tips and tricks for working on the website: http://beam.incubator.apache.org/contribute/contribution-guide/#website On Thu, Dec 1, 2016 at 5:51 AM, Sandeep Deshmukh wrote: > Yes, got it. I found nothing the

Re: [DISCUSS] Graduation to a top-level project

+1 You might even say I'm beaming with pride ;-) On Tue, Nov 22, 2016 at 11:58 AM, Kenneth Knowles wrote: > +1 !!! > > I especially love how the diversity of the community has contributed to the > conceptual growth and quality of Beam. I can't wait for more! > > On Tue, Nov 22, 2016 at 11:22 AM

Re: Compilation Failure: release-0.2.0 referencing sdk-0.3.0

When I was working on the pending quickstart guide [1] earlier today, I found I had to specify -DarchetypeVersion=LATEST [2] during archetype generation to avoid this issue. $ mvn archetype:generate -DarchetypeGroupId=org.apache.beam -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-starter -D

Re: Contributing for improvement of the Beam website

And just FYI, there's a section on the website at the end of the contribution guide: http://beam.incubator.apache.org/contribute/contribution-guide/#website On Sat, Oct 29, 2016 at 9:46 AM, Minudika Malshan wrote: > Hi, > > Here is the PR[1] for BEAM-835. > Please review. > > [1] https://github.

Re: [DISCUSS] Merging master -> feature branch

e-branches should most of the > > time > > >> > be trivial (no conflicts) and a committer should be allowed to > > self-merge > > >> > once tests pass. > > >> > As for conflicts, even for the smallest once I'd go with review

Re: [DISCUSS] Merging master -> feature branch

On Tue, Oct 25, 2016 at 9:44 PM, Jean-Baptiste Onofré wrote: > Agree. When possible it would be great to have the branch merged on master > quickly, even when it's not fully ready. It would give more visibility to > potential contributors. > This thread is about the opposite, I think -- merging

Re: [DISCUSS] Current ongoing work on runners

On Mon, Oct 24, 2016 at 12:52 PM, Ismaël Mejía wrote: > Hello, > > I am really happy to see new runners been contributed to our community > (e.g. GearPump and Apex recently). We have not discussed a lot about the > current capabilities of both runners. > > Following the recent discussion about ma

Re: A CI Server for Beam?

Jason recently started documenting the project's testing setup. See http://beam.incubator.apache.org/contribute/testing/ On Sat, Oct 22, 2016 at 11:18 PM, Jean-Baptiste Onofré wrote: > Hi > > We use both Jenkins and Travis. > > Jenkins is on build.apache.org and you can find Travis yml in source

Re: [ANNOUNCEMENT] New committers!

Wonderful to see your contributions recognized ;-) On Fri, Oct 21, 2016 at 3:46 PM, Ahmet Altay wrote: > Congratulations to all of you! > > Ahmet > > On Fri, Oct 21, 2016 at 3:35 PM, Ben Chambers > > wrote: > > > Congrats. +3! > > > > On Fri, Oct 21, 2016 at 3:34 PM Kenneth Knowles > > wrote:

Re: [PROPOSAL] Introduce review mailing list and provide update on open discussion

Related to #3-5: Also, as we discussed earlier [1], there will be an additional level of tracking in jira for deeper proposal-style conversations to help us keep track of which ones are still under discussion on the dev@ list (which, as usual, remains the source of truth). The details are still in

Re: [REMINDER] Technical discussion on the mailing list

> > At the end of the day, it comes down to two questions: > > 1) Are there technical and project direction discussions happening off > list and not reflected back to the list? > > 2) If yes, are the concrete decisions being made as a result of the off > list discussions? > From an Apache standpo

Re: [PROPOSAL] New Beam website design?

> > > I want to revive this thread if I may. Is there any way I can help on the > > website redesign? Additionally, is anyone currently working on the UI/UX > > design? I want to make sure I don't duplicate any work. > > > > Cheers! > > > > James >

Re: FYI: All Runners Tested In Precommit

Awesome! Strong tests are hugely important in a project with so many diverse components. On Thu, Sep 15, 2016 at 12:16 PM, Jason Kuster wrote: > Hi all, > > Just a quick update -- as of yesterday all new PRs now run the WordCount > end-to-end test against every runner in master (Flink, Spark, Da

Re: JavaDoc

+ Dan Thanks, Jesse. I believe Dan has pending PRs (pull/38) to update the site after 0.2.0. On Thu, Sep 15, 2016 at 8:26 AM, Jesse Anderson wrote: > Only the 0.1.0 JavaDoc is on the website > . It should have > 0.2.0. > > Thanks, > > Jesse

Re: [PROPOSAL] Website page or Jira to host all current proposal discussion and docs

ards > JB > > > > On Aug 10, 2016, 06:09, at 06:09, Frances Perry > wrote: > >So to summarize where I think this thread is at -- we'd like to more > >clearly lay out the expectations for larger proposals. > >- Explain what the design doc / proposal s

Re: [PROPOSAL] Website page or Jira to host all current proposal discussion and docs

> > > > Regards > > > > > > JB > > > > > > > > > > > > > > > > > > On 08/08/2016 10:03 AM, Aljoscha Krettek wrote: > > > > > > > > > > > >> Please have a look at this: > >

Re: [PROPOSAL] Website page or Jira to host all current proposal discussion and docs

I'm a huge fan of keeping all the details related to a topic in a relevant jira issue. On Sun, Aug 7, 2016 at 9:31 PM, Jean-Baptiste Onofré wrote: > Hi guys, > > we have now several technical discussions, sent on the mailing list with > link to document for details. > > I think it's not easy for

Re: Proposal: Dynamic PIpelineOptions

+Amit, Aljoscha, Manu Any comments from folks on the Flink, Spark, or Gearpump runners? On Tue, Aug 2, 2016 at 11:10 AM, Robert Bradshaw < rober...@google.com.invalid> wrote: > Being able to "late-bind" parameters like input paths to a > pre-constructed program would be a very useful feature, an

Re: [REFLECT] Beam’s Half Birthday!

gt; I'd also really like to see Gearpump runner (maybe also Apex) and > Python > > >> SDK in the main branch. > > >> > > >> Thanks, > > >> Dan > > >> > > >> > > >> On Mon, Aug 1, 2016 at 8:36 AM, Aljoscha Kret

Re: [PROPOSAL] New Beam website design?

look ;-) On Thu, Jun 16, 2016 at 10:11 PM, Jean-Baptiste Onofré wrote: > Good point. It make sense to wait that it's actually implemented and > available before putting on the website. > > Thanks ! > Regards > JB > > > On 06/17/2016 07:01 AM, Frances Perry wrote: &g

[REFLECT] Beam’s Half Birthday!

Hi Beamers! It’s been six months today since Beam was accepted into incubation. It’s thrilling how far we’ve come since then! I’d like to volunteer to put together a post on the Beam blog summarizing our progress since February. Here’s a starting point... What am I missing that we should include?

Re: [KUDOS] Contributed runner: Gearpump!

Awesome! On Wed, Jul 20, 2016 at 6:42 PM, Manu Zhang wrote: > Thanks Kenn and others for the review and help along the way. Feel free to > ping me on slack if you want to know more about gearpump-runner or > gearpump. > > Thanks everyone :) > Manu Zhang > > > > On Thu, Jul 21, 2016 at 8:31 AM J

Re: Display Data Runner Support

Perhaps its worth filing jira issues to investigate this integration for other runners? I'm guessing those might be good starter tasks for folks with the right background. On Sun, Jul 3, 2016 at 11:27 PM, Aljoscha Krettek wrote: > Thanks Scott for this compilation of information! I'll look into

Re: Beam Interview

Love this, Jesse! And pretty inspired reading the answers so far ;-) On Mon, Jul 11, 2016 at 1:42 PM, Jesse Anderson wrote: > Thanks! > > On Mon, Jul 11, 2016 at 1:02 PM Ismaël Mejía wrote: > > > Great Idea, I just added my answers, English is not my native language, > so > > feel free to edit

Re: Scala DSL

+Rafal & Andrew again I am leaning DSL for two reasons: (1) scio uses the existing java execution environment (and won't have a language-specific fn harness of its own), and (2) it changes the abstractions that users interact with. I recently saw a scio repl demo from Reuven -- there's some reall

Re: [NOTICE] Change on Filter

Release notes for each release are being tracked in JIRA. For example: https://issues.apache.org/jira/browse/BEAM/fixforversion/12335764/ Davor is planning to send a follow up email about how we use this process. And as we redo the website layout, we should figure out how to surface this informatio

Re: [thread fork] Apache Beam & Google Cloud Dataflow

With my Google employee hat on, I'd like to soften that claim a little ;-) Currently, the Beam SDK runs again Google Cloud Dataflow. But since Beam isn't itself ready for prime time yet, Google doesn't officially provide support for running Beam on Cloud Dataflow right now, and Google Cloud Datafl

Re: [PROPOSAL] New Beam website design?

e Onofré wrote: > Hi Frances, > > great doc ! > > Maybe in the "Learn" section, we can also add IOs (like SDKs, and > runners), like we do in Camel (http://camel.apache.org/components.html) > For the SDKs, I would also add DSLs in the same section. > > WDYT ? &g

Re: [PROPOSAL] New Beam website design?

Good point, JB -- let's redo the page layout as well. I started with your proposal and tweaked it a bit to add in more details and divide things a bit more according to use case (end user vs. runner/sdk developer): https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edi

Re: [VOTE] groupId/artifactId naming & layout

Totally agree on discussing this ;-) I think Davor was just suggesting we lay out all options and understand them before calling for a vote between them. On Fri, Jun 3, 2016 at 10:19 AM, Jean-Baptiste Onofré wrote: > The purpose of the vote is to get a consensus actually. > > We have two options

Re: [DISCUSS] Developing new components -- branches, maturity, and committers

in the main >>>> >>> repo? >>> >>>> I would propose that feature branches live in the repos of the >>>> committers >>>> who are working on a feature. >>>> >>>> On Thu, 19 May 2016 at 11:54 Jean-Baptiste Onofré

Re: Fwd: machine learning API, common models

We could have a module with a library of PTransforms (similar to the join library in extensions) -- so it wouldn't be part of the core / required SDK. On Fri, May 20, 2016 at 3:35 PM, Henry Saputra wrote: > I am a bit concern about adding ML model APIs to Beam because the fluctuate > nature of

[DISCUSS] Developing new components -- branches, maturity, and committers

Hi Beamers -- I’m thrilled by the recent energy and activity on writing new Beam runners! But that also means it’s probably time for us to figure out how, as a community, we want to support this process. ;-) Back near the beginning, we had a thread [1] discussing that feature branches are the pre

Re: (Virtual) Beam Developers Meetup - 5/4 @ 8am PDT

t; Hi Frances, >>> >>> Thanks for setting up the next Dev meeting. I'd like to join and see >>> how everyone is doing. I'll add some content to the slides. >>> >>> Cheers, >>> Max >>> >>> On Mon, May 2, 2016 at 3:06 AM,

Re: (Virtual) Beam Developers Meetup - 5/4 @ 8am PDT

veryone is doing. I'll add some content to the slides. > > Cheers, > Max > > On Mon, May 2, 2016 at 3:06 AM, Frances Perry > wrote: > > Looks like we can do up to 25 participants (since we can start it via a > > Google Apps for Work account) and I expect a bunc

Re: (Virtual) Beam Developers Meetup - 5/4 @ 8am PDT

; venkat...@innerzeal.com> wrote: > In my experience, G+Hangouts has a limitation of 10 attendees which is > lame. You could explore zoom or some free conferencing service. > > Thanks! > > On Fri, Apr 29, 2016 at 9:49 PM Frances Perry > wrote: > > > Yes -- will definitely ai

Re: (Virtual) Beam Developers Meetup - 5/4 @ 8am PDT

0, 2016 2:39 PM, "Frances Perry" wrote: > > > As discussed earlier this month, we're going to try a virtual Beam > meeting > > for anyone who is interested in joining. > > > > *When:* Wednesday 5/4 at 8am PDT > > > > *Where: *Google Hangouts

(Virtual) Beam Developers Meetup - 5/4 @ 8am PDT

As discussed earlier this month, we're going to try a virtual Beam meeting for anyone who is interested in joining. *When:* Wednesday 5/4 at 8am PDT *Where: *Google Hangouts ok by folks? Alternative suggestions? *Agenda: *Here's a list to get us started. Please suggest other things you'd like to

Re: [DISCUSS] Beam IO &runners native IO

> > @Frances Sources are not simple DoFns. They add additional > functionality, e.g. checkpointing, watermark generation, creating > splits. If we want sinks to be portable, we should think about a > dedicated interface. At least for the checkpointing. > We might be mixing sources and sinks in thi

Re: [DISCUSS] Beam IO &runners native IO

+Dan Halperin (who is OOO for a couple of days) Yes, there are plans for unbounded sinks. But unlike sources, sinks don't add any additional functionality beyond a ParDo (they just make it more obvious how to use a ParDo appropriately to get the right fault tolerance). So they haven't been priorit

Re: [PROPOSAL] New sdk languages

t;>>> First congratulations guys, excellent job / API, the scalding >>>>>>>> >>>>>>> touches >>> >>>> are >>>>> >>>>>> pretty neat (as well as the Tap abstraction). I am also new to

Re: getting started?

Here's the Beam jira: https://issues.apache.org/jira/browse/BEAM If you want to dig into examples to learn the Beam model, I'd suggest starting with these walkthroughs: https://cloud.google.com/dataflow/examples/wordcount-example https://cloud.google.com/dataflow/examples/gaming-example (Curr

Re: [PROPOSAL] New sdk languages

+Neville and Rafal for their take ;-) Excited to see this out. Multiple community driven SDKs are right in line with our goals for Beam. On Thu, Mar 24, 2016 at 3:04 PM, Ismaël Mejía wrote: > Addendum: actually the semantic model support is not so far away as I said > before (I havent finished

Re: [PROPOSAL] Create new Jira IO component

ow is IO. >> Maybe we should rename -extensions to -io and then rename sdk-java-gcp to >> sdk-java-io-gcp? >> >> On Mon, Mar 21, 2016 at 9:57 AM, Frances Perry wrote: >> >> The original plan was that IOs would just be in the library extensions >>> (e.g. s

Re: Committer workflow

I'd like to suggest a slightly stronger review process. In the draft guidelines it currently says all PRs should get reviewed, regardless of author. The only real exception to that is something that is an emergency f

Re: [PROPOSAL] Create new Jira IO component

The original plan was that IOs would just be in the library extensions (e.g. sdk-java-extensions). It'd fine to subdivide that further if needed, but maybe we should wait until it gets a bit bigger? Dan, what do you think, as component owner? On Mon, Mar 21, 2016 at 2:33 AM, Jean-Baptiste Onofré

Re: Capability Matrix

That's "partially". Check out the full matrix for complete details: http://beam.incubator.apache.org/capability-matrix/ On Thu, Mar 17, 2016 at 4:50 PM, Jean-Baptiste Onofré wrote: > Great job ! > > By the way, when you use ~ in the matrix, does it mean that it works only > in some cases (depend

Draft Contribution Guide

Hi Beamers! We've started a draft for the Beam contribution guide. Please take a look and provide feedback. Once things settle, we'll get this moved over on to the Beam website. Frances

Re: Optimization of logical plan

; order > > of your transforms but rely on lazy operation and DAG scheduler pipeline. > > > > 2016-02-19 13:20 GMT+08:00 Frances Perry : > > > > > (I'm not familiar with the details of Catalyst itself.) > > > > > > The existing runners (Cloud

Re: Optimization of logical plan

(I'm not familiar with the details of Catalyst itself.) The existing runners (Cloud Dataflow, Spark, Flink) all do optimizations of their own, though it's quite likely there's a set of optimizations that are conceptually shared. For example, something like ParDo fusion is pretty basic to executing

Re: status update

rtifacts are always hosted on downloads page.Maser > >>>> need > >>>> not > >>>> be the one with production ready state. > >>>> 3. It is quite intuitive to use *master* otherwise new > contributors > >

Re: MR runner

Beam is by definition a superset of what can be done in a traditional batch MapReduce-style runner. Although some things like Windowing can be pretty easily mocked in MapReduce, unbounded input collections would be pretty hard to implement without implementing a full micro-batch-based streaming eng

Re: status update

Thanks for all the feedback! Please keep it coming as needed. We've gone ahead and created components matching this structure: https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:components-panel We'll work on transition existing state from Google-inter

Re: Apache Storm Runner for Beam?

As we set up the JIRA, I've created a placeholder issue to track interest in a Storm runner: https://issues.apache.org/jira/browse/BEAM-9 On Wed, Feb 10, 2016 at 9:36 PM, Jean-Baptiste Onofré wrote: > Hi Taylor, > > In a first time, I would say in Beam (as we do for the other runners). > > Later

Re: Apache Storm Runner for Beam?

isting Runners to be > brought > > in into Bean source code repository as internal modules, right? > > > > Thanks, > > > > Henry > > > > On Wed, Feb 10, 2016 at 4:59 PM, Frances Perry > > wrote: > > > >> One of the initial things

Re: Apache Storm Runner for Beam?

One of the initial things we are focusing on as we move things to Beam is cleaning up the API that new runners will implement. (The current version is neither particularly stable nor clean.) But as long as you are comfortable with a little churn, feel free to start poking around in the code that wi

Re: contribution to dataflow

In the meantime, you can learn about the Beam model: * https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 * https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102 and check out the code that will be seeding the donation (after a bit of refactoring): * Dataflow Java SD

Re: status update

, 2016 at 7:46 AM, Frances Perry wrote: > Hi Beamers! > > Here’s the Apache Beam: Technical Vision > <https://docs.google.com/document/d/1UyAeugHxZmVlQ5cEWo_eOPgXNQA1oD-rGooWOSwAqh8/edit?pref=2&pli=1#heading=h.e5s64nliyukh> > document I shared last week with a number o

status update

Hi Beamers! Here’s the Apache Beam: Technical Vision document I shared last week with a number of you. (Now we have a dev@ list to share it more widely -- yay!) I just wanted

Re: PPMC

On Fri, Feb 5, 2016 at 7:23 AM, Jean-Baptiste Onofré wrote: > By the way Tyler, it's the first time that I'm named "beamer" ;) > > Got bunch of "bird names" during rugby games, but never beamer. I'm proud > of this one ;) > Not quite as much fun as "Dataflower", but it'll do ;-)