Re: [DISCUSS] Updating contribution guide for gitbox

2017-11-29 Thread Ismaël Mejía
+1 to Kenneth proposal, using reviewer and asignee, for the merge strategy (a) +1 with the same arguments (preserving commits when they are meaningful and isolated, ask committers to do extra squash if needed. I don't really favor having one big commit per PR (in particular if the change is big) b

Schema-Aware PCollections

2017-11-29 Thread Reuven Lax
There has been a lot of conversation about schemas on PCollections recently. There are a number of reasons for this. Schemas as first-class objects in Beam provide a nice base for building BeamSQL. Spark has provided schema-support via Dataframes for over two years, and it has proved to be very pop

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-29 Thread Pei HE
+1 On Thu, Nov 30, 2017 at 8:48 AM, tarush grover wrote: > +1 (binding). > > On Thu, 30 Nov 2017 at 2:06 AM, Eugene Kirpichov > wrote: > >> +1 (binding). >> >> I also think that the process here was handled in an acceptable fashion. >> Due to the way our infrastructure works, merging to master

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-29 Thread tarush grover
+1 (binding). On Thu, 30 Nov 2017 at 2:06 AM, Eugene Kirpichov wrote: > +1 (binding). > > I also think that the process here was handled in an acceptable fashion. > Due to the way our infrastructure works, merging to master was required in > order to gather essential information for a vote. Thou

Re: [DISCUSS] Thinking about Beam 3.x roadmap and release schedule

2017-11-29 Thread Ahmet Altay
My wishlist for 2018 would be - Python 3 support - Python SDK to work with more runners. This is covered in portability in general. I would like to see an enterprise grade Python SDK that can run on a range of Beam runners. - Related to the above item, full streaming support with Python SDK. - Pyt

How to cope with Maven transient network issues?

2017-11-29 Thread Eugene Kirpichov
Our builds often hit transient Maven network issues, e.g. this one https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/5331/consoleFull 2017-11-29T02:18:02.936 [ERROR] Failed to execute goal on project beam-sdks-java-io-hadoop-jdk1.8-tests: Could not resolve dependencies for project or

Re: On voting and process

2017-11-29 Thread Romain Manni-Bucau
Hi guys I think everyone has a good will and the issue is more coming from the fact that now 2 build systems have to be maintained - and are not symmetrically maintained :(. Will be fixee soon so no need to discuss it much IMHO. Next time a major change will be done the impacts once merged shoul

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-29 Thread Eugene Kirpichov
+1 (binding). I also think that the process here was handled in an acceptable fashion. Due to the way our infrastructure works, merging to master was required in order to gather essential information for a vote. Though I suppose we probably could have had an additional vote about whether or not we

On voting and process

2017-11-29 Thread Robert Bradshaw
[Starting a new thread to not hijack the voting one...] On Wed, Nov 29, 2017 at 10:19 AM, Lukasz Cwik wrote: > I have to disagree about comments about process since in order: > * there was a discussion thread before any POCs were created where Gradle > and Bazel were brought up > * a PR was creat

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-29 Thread Lukasz Cwik
I have to disagree about comments about process since in order: * there was a discussion thread before any POCs were created where Gradle and Bazel were brought up * a PR was created that was brought up on dev@ and available to anyone for comment * on the discussion thread it was specifically broug

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-29 Thread Robert Bradshaw
+1 (binding) I agree with what both JB and Reuven had to say about process. On Wed, Nov 29, 2017 at 7:45 AM, Jean-Baptiste Onofré wrote: > Hi Reuven, > > I know that the merge was not malicious. No problem at all ;) > > It's just about the community and consensus. > > For instance, I did discuss

Re: [discuss] java profile

2017-11-29 Thread Romain Manni-Bucau
Le 29 nov. 2017 18:48, "Lukasz Cwik" a écrit : In practice you can get incremental builds, its not just theory. :) give it a try matching all dev build habits. Current gradle setup already breaks it setting up the whole tree for subprojects, same will apply for incremental setup and is boring

Re: [discuss] java profile

2017-11-29 Thread Lukasz Cwik
In practice you can get incremental builds, its not just theory. On Tue, Nov 28, 2017 at 10:12 PM, Romain Manni-Bucau wrote: > > > Le 29 nov. 2017 01:29, "Robert Bradshaw" a écrit : > > I am curious what you mean by the "Python Build" as by nature there's > no (significant) compilation that hap

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-29 Thread Jean-Baptiste Onofré
Hi Reuven, I know that the merge was not malicious. No problem at all ;) It's just about the community and consensus. For instance, I did discussion + vote to have a consensus about Spark 2 support & upgrade. It's not a big deal, but we have to be careful with our communities (here the dev co

Re: [DISCUSS] Updating contribution guide for gitbox

2017-11-29 Thread Aljoscha Krettek
I think I agree with Kenn on the "merge question": - There should be a merge commit because this records important information, for example, I like having the option of figuring out what PR certain commits came from - Individual meaningful commits of a PR should be preserved, I think having co

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-29 Thread Reuven Lax
Thanks for bringing this up JB. I don't think the merge to master was done maliciously. We don't usually vote before merging PRs, and since that PR did not affect the normal build process. It was done to gather data about how well Gradle works, not to force any one outcome (one possible result of

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-29 Thread Aljoscha Krettek
+1 I agree with JB on the process but I think overall using Gradle will bring only benefits. > On 29. Nov 2017, at 09:44, Jean-Baptiste Onofré wrote: > > -0 > > It's not for the change itself (gradle build is interesting) but for the > process used here, which, IMHO, is not clean. > > My ma

Re: [DISCUSS] Thinking about Beam 3.x roadmap and release schedule

2017-11-29 Thread Ismaël Mejía
It is good to see so much enthusiasm about the future of Beam independently of the fact that we call it Beam 3 or no. I have some doubts about the idea of a release per month, Apache releases are designed to be slow-pace (via the 3-day voting process). It is just a question that we have in the sam

Re: Apache Beam Workshop in Guadalajara Mexico

2017-11-29 Thread Jesse Anderson
That's great! On Wed, Nov 29, 2017, 4:00 AM Etienne Chauchot wrote: > Very nice Griselda! > > Looking forward to get feedback ! > > Thanks > > Etienne > > > Le 29/11/2017 à 09:11, Jean-Baptiste Onofré a écrit : > > Hi Gris, > > > > That's really great ! Thanks for sharing. > > > > By the way, ne

Re: [DISCUSSION] Runner agnostic metrics extractor?

2017-11-29 Thread Jean-Baptiste Onofré
Hi Etienne, yeah, I think it makes sense to update the PoC. I like the package/class name you are proposing. Thanks ! Regards JB On 11/29/2017 10:30 AM, Etienne Chauchot wrote: Thanks Ben for your comments! Indeed, there is an issue about failover regarding the polling thread. To that exten

Re: [DISCUSSION] Runner agnostic metrics extractor?

2017-11-29 Thread Etienne Chauchot
Thanks Ben for your comments! Indeed, there is an issue about failover regarding the polling thread. To that extent, pushing metrics to a sink would be better. To make this push runner agnostic, doing the code in the runner-common part of beam would be good. Maybe in the runner API like JB sug

Re: Apache Beam Workshop in Guadalajara Mexico

2017-11-29 Thread Etienne Chauchot
Very nice Griselda! Looking forward to get feedback ! Thanks Etienne Le 29/11/2017 à 09:11, Jean-Baptiste Onofré a écrit : Hi Gris, That's really great ! Thanks for sharing. By the way, next week I will be at Strata Singapore. I know some beamers will be around. Regards JB On 11/29/201

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-29 Thread Jean-Baptiste Onofré
-0 It's not for the change itself (gradle build is interesting) but for the process used here, which, IMHO, is not clean. My major concern is the fact that gradle build has been merged on master before the vote. I would have start the vote based on the discussion and PR branch (acting as a P

Re: [DISCUSS] Updating contribution guide for gitbox

2017-11-29 Thread Jean-Baptiste Onofré
Hi, I don't see why gitbox merge button change what we are doing. I agree with Kenn for 1 (reviewer field) & 2 (assignee field). IMHO, for 3, I think the reviewer should only use rebase & merge. The squash should be under the contributor scope. The reviewer can ask to squash some commits, but

Re: Apache Beam Workshop in Guadalajara Mexico

2017-11-29 Thread Jean-Baptiste Onofré
Hi Gris, That's really great ! Thanks for sharing. By the way, next week I will be at Strata Singapore. I know some beamers will be around. Regards JB On 11/29/2017 08:31 AM, Griselda Cuevas wrote: Hi Everyone, I wanted to share with you that on December 2nd, Wizeline Academy [1] will host