Jenkins build is still unstable: beam_Release_NightlySnapshot #402

See

[PROPOSAL] Error Reporting Callback

Hello everyone, Currently when using Dataflow, Stackdriver Logs are essentially the first and last line of defence for debugging a job. After a burst of exceptions, stacktraces are actually dropped and finding the source of an error quickly devolves into finding a needle in a haystack. We should

Re: [PROPOSAL] Running Splittable DoFn via Source API

+1 I’m a bit hesitant, though, because stage 3 (in the new plan) could become the current stage 1: now, stage 1 is “waiting for Runners to support SDF” while stage 2 is “implement sources as SDF”. We are blocked by Runner support in stage 1 while in the new scheme we would be blocked on Runner

Re: [PROPOSAL] Running Splittable DoFn via Source API

+1 I just have a little question: are we blocked to move forward for the support in the runners or it's just a question of focus ? I think we could focus on this after the first stable release. Thought ? Regards JB On 05/01/2017 07:22 AM, Eugene Kirpichov wrote: Hey all, TL;DR: Developme

HDFSFileSource and distributed Apex questions

Hi all, Super newb question here - I'm just getting started playing with beam, and wanted to check out its capabilities to run on Apex. So I tried to follow the directions here: https://beam.apache.org/documentation/runners/apex/ The direct

Re: Beam spark 2.x runner status

Spark 2.1.1 has been released. Consider using the new release in this work. Thanks On Wed, Mar 29, 2017 at 5:43 AM, Jean-Baptiste Onofré wrote: > Cool for the PR merge, I will rebase my branch on it. > > Thanks ! > Regards > JB > > > On 03/29/2017 01:58 PM, Amit Sela wrote: > >> @Ted definitel

Future processing time timers and final watermark

Hi, While working on SDF support in the Apex runner, I see the scenario where processing time timers are set in the future. These never trigger, the topology exits with the final watermark before. What is the correct way to handle this? Should the final watermark be held until all processing time

Re: Beam spark 2.x runner status

Hi Ted, My branch used Spark 2.1.0 and I just updated to 2.1.1. As discussed with Aviem, I should be able to create the pull request later today. Regards JB On 05/03/2017 02:50 AM, Ted Yu wrote: Spark 2.1.1 has been released. Consider using the new release in this work. Thanks On Wed, Mar

Re: Let's make Beam transforms comply with PTransform Style Guide

Hey all, The effort is complete: all transforms have been brought in accordance with the style guide and the JIRAs are closed! In nearly all cases the fixes introduced small but backward-incompatible changes, but always with a simple migration path, and I believe the Beam API surface is overall m