Google donated Jenkins slaves

[Daniel Takamori]

Hello Beam Devs!
Today I had a meeting a with Davor Bonaci and Jason Kuster from Google and
was able to add 3 new build slaves to the Apache Jenkins cluster at
builds.apache.org.  While these nodes are for the whole of the foundation's
use, you are more than welcome to use the 'beam' label to tag your Jenkins
jobs to use these slaves directly.
If you have any questions, feel free to e-mail me directly, issue a ticket
on JIRA or find us on HipChat #asfinfra.

Cheers,
-Pono on behalf of the Infrastructure Team

Re: getting started?

[Frances Perry]

Here's the Beam jira:

https://issues.apache.org/jira/browse/BEAM

If you want to dig into examples to learn the Beam model,  I'd suggest
starting with these walkthroughs:

https://cloud.google.com/dataflow/examples/wordcount-example

https://cloud.google.com/dataflow/examples/gaming-example

(Currently they are hosted elsewhere and only describe executing with the
Cloud Dataflow runner, but we will be generalizing them and moving to the
Beam site soon! The corresponding source code is already available in the
Beam repo.)

Frances
On Mar 28, 2016 4:18 PM, "Srikumar Chari" 
wrote:

> JB,
>
> Thanks will review the samples - I am assuming they are under the git repo.
>
> Regarding experience, have been working with Spark mostly, some Flink
> experience and even less Akka streams. Applications are real time/low
> latency evals/transformations and reporting.
>
> Do you have a link to the Jira project?
> —
>
> Thanks
> Sri
>
>
>
>
>
>
>
> On 3/24/16, 1:19 PM, "Jean-Baptiste Onofré"  wrote:
>
> >Hi Sri,
> >
> >you can start with samples, sharing your experience and use case with
> beam.
> >You can also take a look on the Jira and submit PR.
> >
> >Thanks !
> >
> >Regards
> >JB
> >
> >On 03/24/2016 05:47 PM, Srikumar Chari wrote:
> >> Beam team,
> >>
> >> I am very excited about Beam, everything I’ve read about it is awesome.
> >>
> >> I just joined this alias and am interested in becoming a contributor.
> Was wondering if there a natural place for me to start? I’ve mostly trying
> to piece things from a scattering of all the docs/blogs.
> >>
> >> Sorry for rookie question :)
> >> —
> >>
> >> Thanks
> >> Sri
> >>
> >
> >--
> >Jean-Baptiste Onofré
> >jbono...@apache.org
> >http://blog.nanthrax.net
> >Talend - http://www.talend.com
>

Re: [PROPOSAL] Writing More Expressive Beam Tests

[Robert Bradshaw]

On Fri, Mar 25, 2016 at 4:28 PM, Ben Chambers
 wrote:
> My only concern is that in the example, you first need to declare all the
> inputs, then the pipeline to be tested, then all the outputs. This can lead
> to tests that are hard to follow, since what you're really testing is an
> interleaving more like "When these inputs arrive, I get this output. Then
> when this happens, I get that output. Etc.".

+1 to pursuing this direction.

> What if instea of returning a PTransform we had
> a "TestSource".

I think TestSource is a PTransform.

> so we did something like:
>
> TestPipeline p = TestPipeline.create();
> TestSource source = p.testSource();
>
> // Set up pipeline reading from source.
> PCollection sum = ...;

I'm really curious what the "..." looks like. How are we using the source?

> BeamAssert sumAssert = BeamAssert.sum();

Did you mean BeamAssert.that(sum)?

> // Test for the Speculative Pane
> source.addElements(...);
> source.advanceWatermark(...);
> sumAssert.thatWindowPane(...);
>
> // Test for the On Time Pane
> source.addElements(...)
> source.advanceWatermark(...);
> sumAssert.thatWindowPane(...);
>
> etc.

Is there a p.run() at the end?

> We could also allow TestSource to work with multiple input pipelines like
> this:
>
> TestSource intSource = p.testSource(new TypeDescriptor());
> TestSource longSource = p.testSource(new TypeDescriptor());
> ...
> intSource.addElements(...);
> longSource.addElements(...);
> etc.

Would we get at total ordering on the addition of elements/advancement
of watermarks across sources by the temporal ordering of these
operations in the users program (e.g. by incrementing some global
counter)?

> On Fri, Mar 25, 2016 at 4:08 PM Thomas Groh 
> wrote:
>
>> Hey everyone;
>>
>> I'd still be happy to get feedback. I'm going to start working on this
>> early next week
>>
>> Thanks,
>>
>> Thomas
>>
>> On Mon, Mar 21, 2016 at 5:38 PM, Thomas Groh  wrote:
>>
>> > Hey everyone,
>> >
>> > I've been working on a proposal to expand the capabilities of our testing
>> > API, mostly around writing deterministic tests for pipelines that have
>> > interesting triggering behavior, especially speculative and late
>> triggers.
>> >
>> > I've shared a doc here
>> > <
>> https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4/edit?usp=sharing>
>> containing
>> > the proposal and some examples, with world comment access + explicit
>> > committer edit access. I'd welcome any feedback you all have.
>> >
>> > Thanks,
>> >
>> > Thomas
>> >
>>

Re: getting started?

[Srikumar Chari]

JB,

Thanks will review the samples - I am assuming they are under the git repo.

Regarding experience, have been working with Spark mostly, some Flink 
experience and even less Akka streams. Applications are real time/low latency 
evals/transformations and reporting. 

Do you have a link to the Jira project?
— 

Thanks
Sri







On 3/24/16, 1:19 PM, "Jean-Baptiste Onofré"  wrote:

>Hi Sri,
>
>you can start with samples, sharing your experience and use case with beam.
>You can also take a look on the Jira and submit PR.
>
>Thanks !
>
>Regards
>JB
>
>On 03/24/2016 05:47 PM, Srikumar Chari wrote:
>> Beam team,
>>
>> I am very excited about Beam, everything I’ve read about it is awesome.
>>
>> I just joined this alias and am interested in becoming a contributor. Was 
>> wondering if there a natural place for me to start? I’ve mostly trying to 
>> piece things from a scattering of all the docs/blogs.
>>
>> Sorry for rookie question :)
>> —
>>
>> Thanks
>> Sri
>>
>
>-- 
>Jean-Baptiste Onofré
>jbono...@apache.org
>http://blog.nanthrax.net
>Talend - http://www.talend.com