Re: Limit buffer size for a job

[Stephan Ewen]

You can reduce Flink's internal network buffering by adjusting the total
number of network buffers.
See
https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#configuring-the-network-buffers

That should also make a difference.


On Wed, Apr 13, 2016 at 3:57 PM, Andrew Ge Wu 
wrote:

> Thanks guys for the explanation, I will give it a try.
> After all buffers are filled, the back pressure did it’s job, it works so
> far so good, but I will defiantly give a try to control the latency.
>
>
> Thanks again!
>
>
> Andrew
>
>
>
>
> On 11 Apr 2016, at 18:19, Stephan Ewen  wrote:
>
> Hi!
>
> Ufuk's suggestion explains how to buffer less between Flink operators.
>
> Is that what you were looking for, or are you looking for a way to fetch
> more fine grained in the source from the message queue?
> What type of source are you using?
>
> Greetings,
> Stephan
>
>
>
>
> On Mon, Apr 11, 2016 at 5:02 PM, Ufuk Celebi  wrote:
>
>> Hey Andrew,
>>
>> take a look at this here:
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency
>>
>> Does this help?
>>
>> – Ufuk
>>
>> On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu 
>> wrote:
>> > Hi guys
>> >
>> > We have a prioritized queue, where high priority item can jump the
>> queue and we do not want to cache too much record in the buffer.
>> > Is there a way to configure my streaming source to use less buffer? so
>> source always fetch and get latest high prio records?
>> >
>> > Any suggestion? thanks!
>> >
>> >
>> > Andrew
>> > --
>> > Confidentiality Notice: This e-mail transmission may contain
>> confidential
>> > or legally privileged information that is intended only for the
>> individual
>> > or entity named in the e-mail address. If you are not the intended
>> > recipient, you are hereby notified that any disclosure, copying,
>> > distribution, or reliance upon the contents of this e-mail is strictly
>> > prohibited and may be unlawful. If you have received this e-mail in
>> error,
>> > please notify the sender immediately by return e-mail and delete all
>> copies
>> > of this message.
>>
>
>
>
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.
>

Re: Limit buffer size for a job

[Andrew Ge Wu]

Thanks guys for the explanation, I will give it a try.
After all buffers are filled, the back pressure did it’s job, it works so far 
so good, but I will defiantly give a try to control the latency.


Thanks again!


Andrew



> On 11 Apr 2016, at 18:19, Stephan Ewen  wrote:
> 
> Hi!
> 
> Ufuk's suggestion explains how to buffer less between Flink operators.
> 
> Is that what you were looking for, or are you looking for a way to fetch more 
> fine grained in the source from the message queue?
> What type of source are you using?
> 
> Greetings,
> Stephan
> 
> 
> 
> 
> On Mon, Apr 11, 2016 at 5:02 PM, Ufuk Celebi  > wrote:
> Hey Andrew,
> 
> take a look at this here:
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency
>  
> 
> 
> Does this help?
> 
> – Ufuk
> 
> On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu  > wrote:
> > Hi guys
> >
> > We have a prioritized queue, where high priority item can jump the queue 
> > and we do not want to cache too much record in the buffer.
> > Is there a way to configure my streaming source to use less buffer? so 
> > source always fetch and get latest high prio records?
> >
> > Any suggestion? thanks!
> >
> >
> > Andrew
> > --
> > Confidentiality Notice: This e-mail transmission may contain confidential
> > or legally privileged information that is intended only for the individual
> > or entity named in the e-mail address. If you are not the intended
> > recipient, you are hereby notified that any disclosure, copying,
> > distribution, or reliance upon the contents of this e-mail is strictly
> > prohibited and may be unlawful. If you have received this e-mail in error,
> > please notify the sender immediately by return e-mail and delete all copies
> > of this message.
> 


-- 
Confidentiality Notice: This e-mail transmission may contain confidential 
or legally privileged information that is intended only for the individual 
or entity named in the e-mail address. If you are not the intended 
recipient, you are hereby notified that any disclosure, copying, 
distribution, or reliance upon the contents of this e-mail is strictly 
prohibited and may be unlawful. If you have received this e-mail in error, 
please notify the sender immediately by return e-mail and delete all copies 
of this message.

Re: Limit buffer size for a job

[Stephan Ewen]

Hi!

Ufuk's suggestion explains how to buffer less between Flink operators.

Is that what you were looking for, or are you looking for a way to fetch
more fine grained in the source from the message queue?
What type of source are you using?

Greetings,
Stephan




On Mon, Apr 11, 2016 at 5:02 PM, Ufuk Celebi  wrote:

> Hey Andrew,
>
> take a look at this here:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency
>
> Does this help?
>
> – Ufuk
>
> On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu 
> wrote:
> > Hi guys
> >
> > We have a prioritized queue, where high priority item can jump the queue
> and we do not want to cache too much record in the buffer.
> > Is there a way to configure my streaming source to use less buffer? so
> source always fetch and get latest high prio records?
> >
> > Any suggestion? thanks!
> >
> >
> > Andrew
> > --
> > Confidentiality Notice: This e-mail transmission may contain confidential
> > or legally privileged information that is intended only for the
> individual
> > or entity named in the e-mail address. If you are not the intended
> > recipient, you are hereby notified that any disclosure, copying,
> > distribution, or reliance upon the contents of this e-mail is strictly
> > prohibited and may be unlawful. If you have received this e-mail in
> error,
> > please notify the sender immediately by return e-mail and delete all
> copies
> > of this message.
>

Re: Limit buffer size for a job

[Ufuk Celebi]

Hey Andrew,

take a look at this here:
https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency

Does this help?

– Ufuk

On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu  wrote:
> Hi guys
>
> We have a prioritized queue, where high priority item can jump the queue and 
> we do not want to cache too much record in the buffer.
> Is there a way to configure my streaming source to use less buffer? so source 
> always fetch and get latest high prio records?
>
> Any suggestion? thanks!
>
>
> Andrew
> --
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.

Limit buffer size for a job

[Andrew Ge Wu]

Hi guys

We have a prioritized queue, where high priority item can jump the queue and we 
do not want to cache too much record in the buffer.
Is there a way to configure my streaming source to use less buffer? so source 
always fetch and get latest high prio records?

Any suggestion? thanks!


Andrew
-- 
Confidentiality Notice: This e-mail transmission may contain confidential 
or legally privileged information that is intended only for the individual 
or entity named in the e-mail address. If you are not the intended 
recipient, you are hereby notified that any disclosure, copying, 
distribution, or reliance upon the contents of this e-mail is strictly 
prohibited and may be unlawful. If you have received this e-mail in error, 
please notify the sender immediately by return e-mail and delete all copies 
of this message.