unsubscribe

unsubscribe stonnya...@gmail.com

unsubscribe

unsubscribe apur...@google.com

Re: Write bulks files from streaming app

One option (but requires more code): Write to smaller files with frequent triggers to directory_X and once the window properly closes, copy all the files to a single file in your own DoFn. This is certainly more code on your part, but might be worth it. You can use Wait.on() transoform to run your

Re: Pass complex type to FlinkPipelineOptions

You can't set it via the command line. You'll need to programmatically set it. Normally complex types are configured via using a JSON map as shown in this commit: https://github.com/apache/beam/commit/f1d4d41279ad93370a6151902fae6dbd13dde53b#diff-03b47195a404600b2f4576d4ab5fbbb3 This requires the

Re: Generating a window identifier while using a Trigger AfterProcessingTime.pastFirstElementInPane()

Note that @StartBundle is not co-related with a new pane, but an arbitrary runner chosen amount of elements containing any number of windows and elements restricted by the triggering semantics. You can introspect the PaneInfo to see the firing index, index 0 represents the first firing. I don't be

Re: Open Source Challenge, Guadalajara on August 16th.

Hi Arianne, Thanks for the invite ! I would love to be speaker there ! Let's talk private for the logistic details (connection, etc). Thanks again, much appreciated ! Regards JB On 2018-07-19 18:28, Arianne Lisset Navarro Lepe wrote: Forgot to mention, the format is a video-conference ! Reg

Re: Open Source Challenge, Guadalajara on August 16th.

Hello JB,   We have new date (August 16th) for the event to bring on board new contributors in Guadalajara, Mx.   We are looking to have a talk about the Apache Foundation, pretty similar to the one you did in the Apache Beam Summit in San Francisco.   Would you like to be the speaker =) ? Tentativ

Beam SparkRunner and Spark KryoSerializer problem

Folks, Its someone using the SparkRunner out there with the Spark KryoSerializer ? We are being force to use the not so efficient 'JavaSerializer' with Spark because we face the following exception: Exception in thread "main" java.lang.RuntimeException: org.apache.spark.SparkException: Job abor

Generating a window identifier while using a Trigger AfterProcessingTime.pastFirstElementInPane()

Hi Folks, I would like to ask if its possible to be notified when a Windows is created or closed while processing a batch of data. (Sorry for the long post) My scenario: I am using a Session window with a GapDuration of 2 minutes (for testing), during this processing we are assigning a Session id

Write bulks files from streaming app

Hey, I am looking for the advice. I am trying to do a stream processing with Beam on Flink runtime. Reading data from Kafka, doing some processing with it which is not important here and in the same time want to store consumed data to history storage for archive and reprocessing, which is HDFS.

Pass complex type to FlinkPipelineOptions

Hey, I am using Beam with Flink and want to set `stateBacked` via pipeline options available here https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java#L120 The property is an abstract class. I was not able to figure out so