Re: Windowing in a batch pipeline

2017-11-08 Thread Robert Bradshaw
On Wed, Nov 8, 2017 at 5:33 PM, Jacob Marble wrote: > Good evening. I'm trying to nail down windowing. The concept is clear, just > struggling with writing a working pipeline. Tonight the goal is group events > by key and window, in a batch pipeline. All data is "late" because it's a > batch pipel

Windowing in a batch pipeline

2017-11-08 Thread Jacob Marble
Good evening. I'm trying to nail down windowing. The concept is clear, just struggling with writing a working pipeline. Tonight the goal is group events by key and window, in a batch pipeline. All data is "late" because it's a batch pipeline, and I expect nothing to be dropped or processed in a "la

Re: design pattern for enriching data via db lookups?

2017-11-08 Thread Lukasz Cwik
For joining with external data you have some options: * Do direct calls to the external datastore, perform your own in memory caching/expiration. You control exactly what happens and when it happens but as you have done this in the past you know what this entails. * Ingest the external data and per

Re: London Apache Beam meetup 2: call for speakers

2017-11-08 Thread Matthias Baetens
No worries JB, I'll send you a message on how we can plan around this (reschedule the meetup or postpone your session). Thanks for the heads-up, have fun in Singapore! Best, Matthias Op di 7 nov. 2017 om 04:52 schreef Jean-Baptiste Onofré : > Hi, > > unfortunately, I have to decline the invite a

Re: IBM Streams now supports Apache Beam Java applications

2017-11-08 Thread Shen Li
Hi > Do you have IBM specific IOs? Any plans to contribute this or other parts into Apache, or to get more involved with the community? The SDK contains IBM Cloud Object Storage (Swift API) IO, and IBM Streams specific Pub/Sub. There is a newer version of IBM COS coming up using the S3 API. We'd

Re: IBM Streams now supports Apache Beam Java applications

2017-11-08 Thread Kenneth Knowles
This is very cool! I'm curious about the Cloud/Streams specific features - can you give a gist of those? Kenn On Tue, Nov 7, 2017 at 2:45 PM, Daniel Debrunner wrote: > We are excited to announce the release of IBM Streams Runner for Apache > Beam. > > Beam users can submit Beam 2.0 Java applic

Re: IBM Streams now supports Apache Beam Java applications

2017-11-08 Thread Daniel Debrunner
On Tue, Nov 7, 2017 at 4:20 PM, Eugene Kirpichov wrote: > Wow, this is very exciting, thank you for the announcement! It's great to see excitement! :-) > This obviously provokes curiosity, so a few questions: > - What are some features unique to this runner, i.e. features that might > make someb

Re: IBM Streams now supports Apache Beam Java applications

2017-11-08 Thread Ismaël Mejía
Congratulations, this is a nice feature for the IBM Cloud and of course great news for the Apache Beam community. Do you have specific IBM specific IOs? I noticed you guys have an implementation of the OpenStack's Swift FileSystem as part of your SDK. Any plans to contribute this or other parts in