Re: @TearDown guarantees

Le 21 févr. 2018 07:26, "Reuven Lax" a écrit : To close the loop here: Romain, I think your actual concern was that the Javadoc made it sound like a runner could simply decide not to call Teardown. If so, then I agree with you - the Javadoc was misleading (and appears it was

Re: @TearDown guarantees

To close the loop here: Romain, I think your actual concern was that the Javadoc made it sound like a runner could simply decide not to call Teardown. If so, then I agree with you - the Javadoc was misleading (and appears it was confusing to Ismael as well). If a runner destroys a DoFn, it _must_

Re: Beam 2.4.0

I think it's fair to request that the reviewers of these PRs help with your effort to get them merged before the 2.4.0 cut. Existing comments on the PR imply that reviewers think the approaches are reasonable. Assuming that there's not too much work left to be done to address comments, there's a

Re: Beam 2.4.0

Ok In terms of what I'd like included, here is the list: 1. https://github.com/apache/beam/pull/4412 (important to prevent regressions) 2. https://github.com/apache/beam/pull/4674 (can need some more work but can break some api if we do, so current state is a functional trade off). On a more

Re: [YouTube channel] Add video: Apache Beam meetup London 2: use case in finance + IO in Beam and Splittable DoFns

How can people get access to the channel? Thanks, Gaurav On Wed, Feb 21, 2018 at 1:34 PM, Matthias Baetens < matthias.baet...@datatonic.com> wrote: > Hi all, > > This is a proposal to launch the Apache Beam YouTube channel and at the > same time add the first video to the channel. > > We would

[YouTube channel] Add video: Apache Beam meetup London 2: use case in finance + IO in Beam and Splittable DoFns

Hi all, This is a proposal to launch the Apache Beam YouTube channel and at the same time add the first video to the channel. We would like to use the channel to centralize all videos / recordings related to Apache Beam and make it a community driven channel, so people have a one stop shop for

Re: Beam 2.4.0

+1, this is keeping with an every-six weeks cadence. Romain, you can always target Jiras to this release, and then the release manager can decide on a case-by-case basis whether to make sure the fix is included. On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw wrote: >

Re: Beam 2.4.0

+1 on having release trains scheduled. Romain: Do you have a list of PRs that could benefit from increased focus if they want to make it on the upcoming train? On Tue, Feb 20, 2018 at 3:30 PM Ahmet Altay wrote: > +1 for having regular release cycles. Finalizing a release

Re: Beam 2.4.0

+1 for having regular release cycles. Finalizing a release takes time in the order of a few weeks and starting a new release soon after the previous one is a reliable way for having releases every 6 weeks. On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw wrote: > Yep. I am

Re: Beam 2.4.0

Yep. I am starting the "Let's do a 2.4.0 release" thread almost exactly 6 weeks after JB first started the 2.3.0 release thread. On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen wrote: > I would like to +1 the faster release cycle process JB and Robert have been > advocating and

Re: Beam 2.4.0

I would like to +1 the faster release cycle process JB and Robert have been advocating and implementing, and thank JB for releasing 2.3.0 smoothly. When we block for specific features and increase the time between releases, we increase the urgency for PR authors to push for their change to go into

Re: Beam 2.4.0

Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just out so 1 week is a bit fast IMHO. Le 20 févr. 2018 23:13, "Robert Bradshaw" a écrit : > One of the main shifts that I think helped this release was explicitly > not being feature driven, rather releasing

Re: Beam 2.4.0

One of the main shifts that I think helped this release was explicitly not being feature driven, rather releasing what's already in the branch. That doesn't mean it's not a good call to action to try and get long-pending PRs or similar wrapped up. On Tue, Feb 20, 2018 at 2:10 PM, Romain

Re: Beam 2.4.0

There are a lot of long pending PR, would be good to merge them before 2.4. Some are bringing tests for the 2.3 release which can be critical to include. Maybe we should list the pr and jira we want it before picking a date? Le 20 févr. 2018 22:02, "Konstantinos Katsiapis"

Re: Beam 2.4.0

+1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6 (and the latter already has an RC out, so we will likely be blocked on Beam). On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw wrote: > Now that Beam 2.3.0 went out

Re: Scala SDK for Apache Beam

Thanks Eugene. I will look at it. On Tue, Feb 20, 2018 at 12:54 PM, Eugene Kirpichov wrote: > Have you looked at Scio https://github.com/spotify/scio ? > > On Tue, Feb 20, 2018 at 12:50 PM Ankit Jhalaria > wrote: > >> Hey guys, >> >> I am interested

Re: Scala SDK for Apache Beam

Have you looked at Scio https://github.com/spotify/scio ? On Tue, Feb 20, 2018 at 12:50 PM Ankit Jhalaria wrote: > Hey guys, > > I am interested in writing an SDK in Scala for Apache Beam. I was > wondering if someone has already started working on it? If yes, I would >

Beam 2.4.0

Now that Beam 2.3.0 went out (and in record time, kudos to all that made this happen!) It'd be great to keep the ball rolling for a similarly well-executed 2.4. A lot has gone in [1] since we made the 2.3 cut, and to keep our cadence up I would propose a time-based cut date early next week (say

Scala SDK for Apache Beam

Hey guys, I am interested in writing an SDK in Scala for Apache Beam. I was wondering if someone has already started working on it? If yes, I would love to know how to contribute? If work hasn't been started on the Scala SDK, what would be some good pointers/tips to start an implementation?

Re: Code reviews in Beam

I do think we need something less generic than our current @Experimental. I also like the idea of a separate package for unvetted contributions (though Incubating might simply be confusing, given that Apache uses Incubating for something else). Good idea to have a standard way of marking such

Re: Code reviews in Beam

Thanks, Reuven, for bringing this up. This is an area well worth investing time in. Specific comments below. On Mon, Feb 19, 2018 at 10:32 AM, Reuven Lax wrote: > Pedantic > > Overly-pedantic comments (change variable names, etc.) can be frustrating. > The PR author can feel

Re: Code reviews in Beam

I'm ambivalent about the placement of less-stable contributions, but I'd be very much in favor of being clear about how stable a given API is. There's several axes of stability here, too: - Is the API going to stay compatible - Is the implementation going to stay compatible via pipeline updates -

Re: beam shade itself?

This comes from keeping forbidden deps off the API surface. It is probably overkill, but I cannot recall the details. On Tue, Feb 20, 2018 at 8:42 AM, Romain Manni-Bucau wrote: > Hi guys, > > is it intended beam shades itself? > > $ jar tf

Re: Code reviews in Beam

On further thought I like the separate package even more. It allows us to easily isolate all those tests, and not block commits or releases on them. On Tue, Feb 20, 2018 at 7:45 AM, Reuven Lax wrote: > Another idea: we could have a special package for these "unrefined" >

Re: Jenkins job beam_PreCommit_Python_MavenInstall” still fails

I would wait a feedback from the original author of the commit. Regards JB Le 20 févr. 2018 à 16:42, à 16:42, Alexey Romanenko a écrit: >Yes, I switched back to BundleBasedDirectRunner as default direct >runner on current master and now it passes. >So, my question was

Re: Jenkins job beam_PreCommit_Python_MavenInstall” still fails

Yes, I switched back to BundleBasedDirectRunner as default direct runner on current master and now it passes. So, my question was rather if we need to revert this or someone, who aware of this change, can take a look and fix it properly? WBR, Alexey > On 20 Feb 2018, at 16:20, Jean-Baptiste

Re: Code reviews in Beam

I would add some @ToBeRefined or so  Le 20 févr. 2018 à 16:35, à 16:35, Reuven Lax a écrit: >Hi JB, > >You're right, I was thinking more about changes to core when talking >about >the technical-excellence bar. > >I think there still needs to be some bar for new features and

Re: Code reviews in Beam

Fully agree ! Thanks Reuven Regards JB Le 20 févr. 2018 à 16:35, à 16:35, Reuven Lax a écrit: >Hi JB, > >You're right, I was thinking more about changes to core when talking >about >the technical-excellence bar. > >I think there still needs to be some bar for new features and

Re: Code reviews in Beam

Hi JB, You're right, I was thinking more about changes to core when talking about the technical-excellence bar. I think there still needs to be some bar for new features and extension, but I also think it can be much lower (as nobody is breaking anything by merging this). An example of where we

Re: Jenkins job beam_PreCommit_Python_MavenInstall” still fails

Yes it seems to be related to the change on the Python direct runner. Did you try to revert the change to see if it fixes the build ? Thanks Regards JB Le 20 févr. 2018 à 16:13, à 16:13, Alexey Romanenko a écrit: >Hi all, > >Jenkins job

Jenkins job beam_PreCommit_Python_MavenInstall” still fails

Hi all, Jenkins job “beam_PreCommit_Python_MavenInstall” has been constantly failing for last 3 days. Last successful build (#3052) has been produced at Feb 16: https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/lastSuccessfulBuild/

Re: force the coder for a pardo

Agree. It makes sense to me in order to be consistent between the PTransforms. Regards JB Le 20 févr. 2018 à 16:08, à 16:08, Eugene Kirpichov a écrit: >Something similar was discussed a while ago, and lead to the suggestion >in >PTransform Style Guide:

Re: force the coder for a pardo

Something similar was discussed a while ago, and lead to the suggestion in PTransform Style Guide: https://beam.apache.org/contribute/ptransform-style-guide/#setting-coders-on-output-collections This suggestion is currently not followed by ParDo, but your plan moves in that direction, so +1 to

Re: Code reviews in Beam

+1. It's a fair idea to have dedicated guides. Regards JB Le 20 févr. 2018 à 14:43, à 14:43, Alexey Romanenko a écrit: >Reuven, thank you for bringing this topic. > >As a new contributor to Beam codebase I raise two my hands for such >guideline document and I'd

Re: Code reviews in Beam

Hi Reuven I agree with all your points except maybe in term of bar level, especially on new features (like extensions or IOs). If the PRs on the core should be heavily reviewed, I'm more in favor of merging the PR pretty fast even if not perfect. It's not a technical topic, it's really a

Re: Code reviews in Beam

Reuven, thank you for bringing this topic. As a new contributor to Beam codebase I raise two my hands for such guideline document and I'd propose to add it as a new guide into section “Other Guides” on web site documentation. For sure, there are already several very helpful and detailed

Re: Code reviews in Beam

This is excellent! I can't really add anything right now but I think having a PR dashboard is one of the most important points because it also indirectly solves "Review Latency" and "Code Review Response SLA" by making things more visible. -- Aljoscha > On 19. Feb 2018, at 19:32, Reuven Lax

Re: force the coder for a pardo

Not on the PCollection ? Only ParDo ? Le 20 févr. 2018 à 10:50, à 10:50, Romain Manni-Bucau a écrit: >Hi guys, > >any objection to allow to pass with the pardo a coder? Idea is to avoid >to >have to write your own transform to be able to configure the coder when >you

force the coder for a pardo

Hi guys, any objection to allow to pass with the pardo a coder? Idea is to avoid to have to write your own transform to be able to configure the coder when you start from a dofn and just do something like ParDo.of(new MyFn(), new MyCoder()) which is directly integrable into a pipeline properly.

Build failed in Jenkins: beam_Release_NightlySnapshot #691

See Changes: [lcwik] [BEAM-3690] swapping to use mockito-core, hamcrest-core and [github] Updates javadocs of Setup and Teardown -- [...truncated 3.09 MB...]