Re: Direct runner doesn't seem to finalize checkpoint "quickly"

2017-05-11 Thread Jean-Baptiste Onofré
Hi, I moved forward a bit on this. Actually, the IOs seem to work fine from a reading perspective. My issue is more on window/trigger in the pipeline. First, I used FixedWindows with the default trigger (event-time based). With this windows, it seems the trigger is never executed (I have to c

Re: Direct runner doesn't seem to finalize checkpoint "quickly"

2017-05-10 Thread Jean-Baptiste Onofré
Thanks Thomas, Let me try what you are proposing. I keep you posted. Regards JB On 05/10/2017 06:33 PM, Thomas Groh wrote: I'm going to start with number two, because it's got an easy answer: When performing an unbounded read, the DirectRunner will finalize a checkpoint after it has completed

Re: Direct runner doesn't seem to finalize checkpoint "quickly"

2017-05-10 Thread Thomas Groh
I'm going to start with number two, because it's got an easy answer: When performing an unbounded read, the DirectRunner will finalize a checkpoint after it has completed a subsequent read from that split where at least one element was read. A bounded read from an unbounded source will never be fin

Direct runner doesn't seem to finalize checkpoint "quickly"

2017-05-10 Thread Jean-Baptiste Onofré
Hi Beamers, I'm working on some fixes in the JmsIO and MqttIO. Those two IOs behave in a similar way on the reading side: - they consume messages from a JMS or MQTT broker - the "pending" messages are stored in the checkpoint mark. When a new message is added to the checkpoint, we compare the