Re: RMQSource synchronous message ack

2019-03-11 Thread gcandal
First of all, thanks for your time and quick response. I'm not completely sure I understood your example, but is this what you mean: - Sink processes A, B, C - Checkpoint persisted with A, B, C - Notify checkpoint starts - Notify checkpoints ACKs A - Notify checkpoints ACKs B - Job crashes - Job

Re: RMQSource synchronous message ack

2019-03-06 Thread Chesnay Schepler
The acknowledgement has to be synchronous since Flink assume that after notifyCheckpointComplete() all data has been persisted to external systems. For example, if record 1 to 100 were passed to the sink and a checkpoint occurs and completed, on restart Flink would continue with record 101. But

RMQSource synchronous message ack

2019-03-05 Thread Gabriel Candal
Hi, Recently I've opened a Stack Overflow question about latency spikes (~500ms) after a checkpoint operation, even though the operation itself was relatively fast (~50ms). I've come to realize that the c