Re: Dealing with latency in Sink

2017-02-07 Thread Robert Metzger
Hi Mohit, Flink doesn't allow dynamic up or downscaling of parallel operator instances at runtime. However, you can stop and restore from a savepoint with a different parallelism. This way, you can adopt to workload changes. Flink's handling of backpressure is very implicit. If you want to thrott

Dealing with latency in Sink

2017-02-06 Thread Mohit Anchlia
What is the best way to dynamically adapt and tune down number of tasks created to write/read to a sink when sink slows down or the latency to sink increases? I am looking at the sink interface but don't see a way to influence flink to reduce the number of tasks or throttle the volume down to the s