Re: Throttling/effective back-pressure on a Kafka sink

2019-05-30 Thread Derek VerLee
Was any progress ever made on this?  We have seen the same issue in the past.  What I do remember is, whatever I set max.block.ms to, is when the job crashes. I am going to attempt to reproduce the issue again and will report back. On 3/28/19

Re: Throttling/effective back-pressure on a Kafka sink

2019-03-28 Thread Konstantin Knauf
Hi Marc, the Kafka Producer should be able to create backpressure. Could you try to increase max.block.ms to Long.MAX_VALUE? The exceptions you shared for the failure case don't look like the root causes of the problem. Could you share the full stacktraces or even full logs for this time frame.

Throttling/effective back-pressure on a Kafka sink

2019-03-28 Thread Marc Rooding
Hi We’ve got a job producing to a Kafka sink. The Kafka topics have a retention of 2 weeks. When doing a complete replay, it seems like Flink isn’t able to back-pressure or throttle the amount of messages going to Kafka, causing the following error: