Re: [DISCUSS] State of the project: Culture and governance

On Tue, Jan 23, 2018 at 8:44 AM Ismaël Mejía wrote: > - Clear guidelines for the criteria to earn commitership/PMC status. > The PMC discussed this quite a bit and we have now added this to the web site: https://beam.apache.org/contribute/become-a-committer/ Kenn

Re: [DISCUSS] State of the project: Culture and governance

bq. 1. Propose a pull request to their fix branch. This is my favorite and I've mentioned it. Everything is straightforward and explicit. The above makes sense. When the contributor merges the reviewer's pull request, it signifies their willingness to adopt the suggestion, making the combined

Re: [DISCUSS] State of the project: Culture and governance

Le 4 févr. 2018 19:53, "Kenneth Knowles" a écrit : On Wed, Jan 31, 2018 at 1:11 AM, Ismaël Mejía wrote: > > For example if a > first-time contributor fixes some error and has the expected quality > we should accept it quickly, not being picky about some

Re: [DISCUSS] State of the project: Culture and governance

On Wed, Jan 31, 2018 at 1:11 AM, Ismaël Mejía wrote: > > For example if a > first-time contributor fixes some error and has the expected quality > we should accept it quickly, not being picky about some other part > that can be improved that was discovered during the review,

Re: [DISCUSS] State of the project

Le 31/01/2018 à 15:50, Kenneth Knowles a écrit : On Wed, Jan 31, 2018 at 3:40 AM, Etienne Chauchot > wrote: Thanks Kenn and Luke for your comments. WDYT about my proposition (bellow) to add methods to the runner api to

Re: [DISCUSS] State of the project

On Wed, Jan 31, 2018 at 3:40 AM, Etienne Chauchot wrote: > Thanks Kenn and Luke for your comments. > > WDYT about my proposition (bellow) to add methods to the runner api to > enhance the coherence between the runners? > If I understand your point, I think I agree but maybe

Re: [DISCUSS] State of the project

Thanks Kenn and Luke for your comments. WDYT about my proposition (bellow) to add methods to the runner api to enhance the coherence between the runners? WDYT about my other proposition (bellow) of trying to avoid having validates runner tests that are specific to a runner like we have now?

Re: [DISCUSS] State of the project: Culture and governance

Sorry, but I fail to see the difference between Beam and other Apache project. It's not a github project: Beam follows the rules defined by the Apache Software Foundation. Beam website/contribution guide or whatever should just point to Apache website resources. It's clearly explained and

Re: [DISCUSS] State of the project: Culture and governance

Some extra comments inlined: JB, > 1. The Apache Beam website contains a link to the Apache Code of Conduct (on > the dropdown menu under the feather icon ;)). Maybe we can also add a link to > the contribution guide as it's really important, especially for people who > target the

Re: [DISCUSS] State of the project: Feature roadmap for 2018

I think I agree with this, but wanted to point out a few things: 1. High-level DSLs may target the IL directly, rather than going through the high-level PL libraries. This would allow them to make more direct use of the capabilities of the IL. 2. I agree that the portability work is basically

Re: [DISCUSS] State of the project: Feature roadmap for 2018

On Tue, Jan 30, 2018 at 11:25 AM Kenneth Knowles wrote: > I've got some thoughts :-) > > Here is how I see the direction(s): > > - Requirements to be relevant: known scale, SQL, retractions (required > for correct answers) > - Core value-add: portability! I don't know that

Re: [DISCUSS] State of the project: Feature roadmap for 2018

On Tue, Jan 30, 2018 at 11:44 AM, Kenneth Knowles wrote: > (just dev@) > > *Low-level IL* > I wanted to comment more on the common intermediate layer idea of Ben's. > This is an awesome idea but I'm not sure it is Beam so much as Tez or Onyx. > I imagine most runners have some

Re: [DISCUSS] State of the project: Feature roadmap for 2018

(just dev@) *Low-level IL* I wanted to comment more on the common intermediate layer idea of Ben's. This is an awesome idea but I'm not sure it is Beam so much as Tez or Onyx. I imagine most runners have some such representation internally. Our layers in the stack with a low-level IL: 6.

Re: [DISCUSS] State of the project: Feature roadmap for 2018

I've got some thoughts :-) Here is how I see the direction(s): - Requirements to be relevant: known scale, SQL, retractions (required for correct answers) - Core value-add: portability! I don't know that there is any other project ambitiously trying to run Python and Go on "every" data

Re: [DISCUSS] State of the project: Culture and governance

I have to -1 reductions in the code review quality bar as this leads to test problems, which leads to CI issues, which leads to gaps in coverage and then to delayed, bad and broken releases. +1 on converting Google docs to either markdown or including them on the website since it is a valuable

Re: [DISCUSS] State of the project: Culture and governance

Hello again, Some comments inlined: JB, > 1. The code of conduct is the one from Apache. Yes I am not necessarily saying that we need a new one, I am just saying that we need to make this explicit, not sure everybody is aware of it https://www.apache.org/foundation/policies/conduct.html > 2.

Re: [DISCUSS] State of the project: Feature roadmap for 2018

1) Instead of enabling it easier to write features I think more users would care about being able to move their pipeline between different runners and one of the key missing features is dynamic work rebalancing in all runners (except Dataflow). Also, portability is meant to help make a crisp line

Re: [DISCUSS] State of the project

I also think that at a high level the success of Beam as a project/community and as a piece of software depends on having multiple viable runners with healthy set of users and contributors. The pieces that are missing to me: *User-focused comparison of runners (and IOs)* +1 to Jesse's. Automated

Re: [DISCUSS] State of the project

Etienne, for the cross runner coherence, the portability framework is attempting to create an API across all runners for job management and job execution. A lot of work still needs to be done to define and implement these APIs and migrate runners and SDKs to support it since the current set of

Re: [DISCUSS] State of the project

Hi all, Does anyone have comments about my point about dev coherence across the runners? Thanks Etienne Le 22/01/2018 à 16:16, Etienne Chauchot a écrit : Thanks Davor for bringing this discussion up! I particularly like that you listed the different areas of improvement and proposed to

Re: [DISCUSS] State of the project: Community growth

On Tue, Jan 23, 2018 at 4:50 PM, Robert Bradshaw wrote: > +1 to having a "beginner" label. Actually, we have one, but it's > hardly used: https://issues.apache.org/jira/browse/BEAM-1565?jql= > project%20%3D%20BEAM%20AND%20labels%20%3D%20beginner% >

Re: [DISCUSS] State of the project: Culture and governance

On Tue, Jan 23, 2018 at 9:29 AM, Romain Manni-Bucau wrote: > Hi Ismael, > > More you add policies and rules around a project more you need energy to > make it respected and enforced. At that stage you need to ask yourself if it > does worth it? +1. I also agree with JB

Re: [DISCUSS] State of the project: Community growth

+1 to having a "beginner" label. Actually, we have one, but it's hardly used: https://issues.apache.org/jira/browse/BEAM-1565?jql=project%20%3D%20BEAM%20AND%20labels%20%3D%20beginner%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC and is hard to discover. We should probably link to the Jira

Re: [DISCUSS] State of the project: Culture and governance

Great starting points! I agree w/ JB's point that some of this should be private@ discussion. I am keeping my comments to things that I think everyone on dev@ should be part of. For all mentions of ASF policy: we should have our own link to the relevant policy. It is hard enough for us to find

Re: [DISCUSS] State of the project: Culture and governance

Hi Ismael, More you add policies and rules around a project more you need energy to make it respected and enforced. At that stage you need to ask yourself if it does worth it? I'm not sure it does for Beam and even if sometimes on PR you can find some comments "picky" (and guess me I thought it

Re: [DISCUSS] State of the project: Culture and governance

Hi, I would like to remind: we are an Apache project, not an isolated one. As an Apache member, it's really important to me. 1. The code of conduct is the one from Apache. 2. If it's not happen on the mailing list, it doesn't exist. That's the Apache rule. We already discussed about that in the

[DISCUSS] State of the project: Culture and governance

This is a sub-thread of the state of the project one initiated by Davor. Since this subject can be part of the community issues I would like to focus on the state of the project for its contributors so we don’t mix the discussion with the end-user thread. I hope other members of the community

Re: [DISCUSS] State of the project: Feature roadmap for 2018

Hi Ben, about the "technical roadmap", we have a thread about "Beam 3.x roadmap". It already provides ideas for points 3 & 4. Regards JB On 01/22/2018 09:15 PM, Ben Chambers wrote: > Thanks Davor for starting the state of the project discussions [1]. > > > In this fork of the state of the

[DISCUSS] State of the project: Feature roadmap for 2018

Thanks Davor for starting the state of the project discussions [1]. In this fork of the state of the project discussion, I’d like to start the discussion of the feature roadmap for 2018 (and beyond). To kick off the discussion, I think the features could be divided into several areas, as

Re: [DISCUSS] State of the project

Thanks Davor for bringing this discussion up! I particularly like that you listed the different areas of improvement and proposed to assign people based on their tastes. I wanted to add a point about consistency across runners, but through the dev point of view: I've been working on a

Re: [DISCUSS] State of the project

Thanks Davor for opening this discussion and HUGE +1 to do this every year or in cycles. I will fork this thread into a new one for the Culture / Project management part issues as suggested. About the diversity of users across runners subject I think that this requires more attention to

Re: [DISCUSS] State of the project: Community growth

Some ideas I have around the issues mentioned before: * Annual User Survey One important thing we should do is some sort of annual survey of users to have some feedback on the state of the Beam from the users point of view, we can take for example as a template the survey done by the Rust

Re: [DISCUSS] State of the project: Community growth

Think that Beam's great, so happy to grow community -- by my individual involvement and to encourage others. 1). Users: I think that this can follow from the others being done well 2). Contributors: some other projects mark very simple contributions for newcomers to find as onramping. Hadn't

[DISCUSS] State of the project: Community growth

*Hi Everyone, Thanks Davor for starting the discussion around the state of the Apache Beam in 2017[1]. In this fork of that conversation, I’d like to continue the dialogue around how should we grow our community in alignment to the project's vision, goals & values.Here are some areas and prompts

Re: [DISCUSS] State of the project

I think a focus on the runners is what's key to Beam's adoption. The runners are the foundation on which Beam sits. If the runners don't work properly, Beam won't work. A focus on improved unit tests is a good start, but isn't what's needed. Compatibility matrices will help see how your runner of

Re: [DISCUSS] State of the project

bq. are hard to detect in our unit-test framework Looks like more integration tests would help discover bug / regression more quickly. If committer reviewing the PR has concern in this regard, the concern should be stated on the PR so that the contributor (and reviewer) can spend more time in

Re: [DISCUSS] State of the project

I agree with the sentiment, but I don't completely agree with the criteria. I think we need to be much better about reviewing PRs. Some PRs languish for too long before the reviewer gets to it (and I've been guilty of this too), which does not send a good message. Also new PRs sometimes languish

Re: [DISCUSS] State of the project

Hi Davor, Thanks a lot for this e-mail. I would like to emphasize two areas where we have to improve: 1. Apache way and community. We still have to focus and being dedicated on our communities (both user & dev). Helping, encouraging, growing our communities is key for the project. Building

Re: [DISCUSS] State of the project

Hi Davor, I wonder what you mean by the thought of prioritizing diversity across runners. * That there is not full parity across runners: https://beam.apache.org/documentation/runners/capability-matrix/ * That there is an imbalance of users relying on specific runners (Haven't seen any usage

[DISCUSS] State of the project

Hi everyone -- Apache Beam was established as a top-level project a year ago (on December 21, to be exact). This first anniversary is a great opportunity for us to look back at the past year, celebrate its successes, learn from any mistakes we have made, and plan for the next 1+ years. I’d like