Re: Limit buffer size for a job

You can reduce Flink's internal network buffering by adjusting the total number of network buffers. See https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#configuring-the-network-buffers That should also make a difference. On Wed, Apr 13, 2016 at 3:57 PM, Andrew Ge Wu

Re: Limit buffer size for a job

Thanks guys for the explanation, I will give it a try. After all buffers are filled, the back pressure did it’s job, it works so far so good, but I will defiantly give a try to control the latency. Thanks again! Andrew > On 11 Apr 2016, at 18:19, Stephan Ewen wrote: > >

Re: Limit buffer size for a job

Hi! Ufuk's suggestion explains how to buffer less between Flink operators. Is that what you were looking for, or are you looking for a way to fetch more fine grained in the source from the message queue? What type of source are you using? Greetings, Stephan On Mon, Apr 11, 2016 at 5:02 PM,

Re: Limit buffer size for a job

Hey Andrew, take a look at this here: https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency Does this help? – Ufuk On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu wrote: > Hi guys > > We have a prioritized queue, where

Limit buffer size for a job

Hi guys We have a prioritized queue, where high priority item can jump the queue and we do not want to cache too much record in the buffer. Is there a way to configure my streaming source to use less buffer? so source always fetch and get latest high prio records? Any suggestion? thanks!