Re: Flink Kafka ordered offset commit & unordered processing

2019-07-03 Thread Piotr Nowojski
ords and all >> checkpoint succeeded, no back pressure was incurred. >> >> In this case, if some machines crash, how does Flink recover the stalled >> offsets? Which checkpoint does Flink rollback to? I understand that >> commiting offset back to Kafka is merely to show p

Re: Flink Kafka ordered offset commit & unordered processing

2019-07-02 Thread Piotr Nowojski
Hi, If your async operations are stalled, this will eventually cause problems. Either this will back pressure sources (the async’s operator queue will become full) or you will run out of memory (if you configured the queue’s capacity too high). I think the only possible solution is to either

Flink Kafka ordered offset commit & unordered processing

2019-06-29 Thread wang xuchen
Hi Flink experts, I am prototyping a real time system that reads from Kafka source with Flink and calls out to an external system as part of the event processing. One of the most important requirements are read from Kafka should NEVER stall, even in face of some async external calls slowness