Thanks a bunch Stephen, this looks like exactly what I need. I'll post
follow up questions if I have any.
Thanks again,
Parker Coleman
On Wed, Apr 19, 2017 at 1:41 PM, Stephen Sisk wrote:
> I believe that GroupIntoBatches[1] may also be of help and it's available
> right now.
>
> Hope that hel
I believe that GroupIntoBatches[1] may also be of help and it's available
right now.
Hope that helps,
Stephen
[1]
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java
On Wed, Apr 19, 2017 at 1:07 PM Parker Coleman wrote:
>
We're writing text (JSON fields, one per line).
We've written the DoFn; but our issue is that dozens of instances per
window are being generated for a relatively small amount of data. I
haven't look into it, but it might be as many as one DoFn object per record.
Some psuedo code explaining how w
Hi Parker,
What's the format you want to write on Azure ?
Right now, I would say the easiest way is to write your own DoFn that writes on
Azure.
By the way, we are working on a complete IOChannelFactory refactoring with Beam
filesystems. It means that we will be able to write a Beam AzureFil
Sorry if this is a double-post, I tried using the web client but didn't see
the post show up, so instead I've subscribed to the list and am sending
this via my email client.
We're working on an apache beam pipeline that gathers messages from various
pub/sub topics, decodes them, bundles them into
