Re: Apache Flink - Throttling stream flow

2019-11-29 Thread M Singh 
 
Thanks Rong for your references. 
>From what I can see, the rate limiter is initialized statically.  But if the 
>load on downstream services varies, is there a way to update the rater limiter 
>at runtime ?  Please let me know if I missed anything.
Thanks again for your advice.On Wednesday, November 27, 2019, 11:32:06 AM 
EST, Rong Rong  wrote:  
 
 Hi Mans,
is this what you are looking for [1][2]?
--Rong
[1] https://issues.apache.org/jira/browse/FLINK-11501[2] 
https://github.com/apache/flink/pull/7679
On Mon, Nov 25, 2019 at 3:29 AM M Singh  wrote:

 Thanks Ciazhi & Thomas for your responses.
I read the throttling example but want to see if that work with a distributed 
broker like Kinesis and how to have throttling feedback to the Kinesis source 
so that it can vary the rate without interfering with watermarks, etc.
Thanks again 
Mans

On Monday, November 25, 2019, 05:55:21 AM EST, Thomas Julian 
 wrote:  
 
 related

https://issues.apache.org/jira/browse/FLINK-13792

Regards,
Julian.


 On Mon, 25 Nov 2019 15:25:14 +0530 Caizhi Weng  
wrote 


Hi,

As far as I know, Flink currently doesn't have a built-in throttling function. 
You can write your own user-defined function to achieve this. Your function 
just gives out what it reads in and limits the speed it gives out records at 
the same time.

If you're not familiar with user-defined functions, see 
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/udfs.html

Here is a throttling iterator example: 
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java

M Singh  于2019年11月25日周一 上午5:50写道：


Hi:

I have an Flink streaming application that invokes  some other web services.  
However the webservices have limited throughput.  So I wanted to find out if 
there any recommendations on how to throttle the Flink datastream so that they 
don't overload the downstrream services.  I am using Kinesis as source and sink 
in my application.

Please let me know if there any hooks available in Flink, what are the patterns 
that can be used and what are the best practices/pitfalls for using them.

Thanks 
Mans

Re: Apache Flink - Throttling stream flow

2019-11-27 Thread Rong Rong 
Hi Mans,

is this what you are looking for [1][2]?

--
Rong

[1] https://issues.apache.org/jira/browse/FLINK-11501
[2] https://github.com/apache/flink/pull/7679

On Mon, Nov 25, 2019 at 3:29 AM M Singh  wrote:

> Thanks Ciazhi & Thomas for your responses.
>
> I read the throttling example but want to see if that work with a
> distributed broker like Kinesis and how to have throttling feedback to the
> Kinesis source so that it can vary the rate without interfering with
> watermarks, etc.
>
> Thanks again
>
> Mans
>
>
> On Monday, November 25, 2019, 05:55:21 AM EST, Thomas Julian <
> thomasjul...@zoho.com> wrote:
>
>
> related
>
> https://issues.apache.org/jira/browse/FLINK-13792
>
> Regards,
> Julian.
>
>
>  On Mon, 25 Nov 2019 15:25:14 +0530 *Caizhi Weng
> >* wrote 
>
> Hi,
>
> As far as I know, Flink currently doesn't have a built-in throttling
> function. You can write your own user-defined function to achieve this.
> Your function just gives out what it reads in and limits the speed it gives
> out records at the same time.
>
> If you're not familiar with user-defined functions, see
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/udfs.html
>
> Here is a throttling iterator example:
> https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java
>
> M Singh  于2019年11月25日周一 上午5:50写道：
>
> Hi:
>
> I have an Flink streaming application that invokes  some other web
> services.  However the webservices have limited throughput.  So I wanted to
> find out if there any recommendations on how to throttle the Flink
> datastream so that they don't overload the downstrream services.  I am
> using Kinesis as source and sink in my application.
>
> Please let me know if there any hooks available in Flink, what are the
> patterns that can be used and what are the best practices/pitfalls for
> using them.
>
> Thanks
>
> Mans
>
>
>
>

Re: Apache Flink - Throttling stream flow

2019-11-25 Thread M Singh 
 Thanks Ciazhi & Thomas for your responses.
I read the throttling example but want to see if that work with a distributed 
broker like Kinesis and how to have throttling feedback to the Kinesis source 
so that it can vary the rate without interfering with watermarks, etc.
Thanks again 
Mans

On Monday, November 25, 2019, 05:55:21 AM EST, Thomas Julian 
 wrote:  
 
 related

https://issues.apache.org/jira/browse/FLINK-13792

Regards,
Julian.


 On Mon, 25 Nov 2019 15:25:14 +0530 Caizhi Weng  
wrote 


Hi,

As far as I know, Flink currently doesn't have a built-in throttling function. 
You can write your own user-defined function to achieve this. Your function 
just gives out what it reads in and limits the speed it gives out records at 
the same time.

If you're not familiar with user-defined functions, see 
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/udfs.html

Here is a throttling iterator example: 
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java

M Singh  于2019年11月25日周一 上午5:50写道：


Hi:

I have an Flink streaming application that invokes  some other web services.  
However the webservices have limited throughput.  So I wanted to find out if 
there any recommendations on how to throttle the Flink datastream so that they 
don't overload the downstrream services.  I am using Kinesis as source and sink 
in my application.

Please let me know if there any hooks available in Flink, what are the patterns 
that can be used and what are the best practices/pitfalls for using them.

Thanks 
Mans

Re: Apache Flink - Throttling stream flow

2019-11-25 Thread Thomas Julian 
related



https://issues.apache.org/jira/browse/FLINK-13792



Regards,

Julian.







 On Mon, 25 Nov 2019 15:25:14 +0530 Caizhi Weng  
wrote 



Hi,



As far as I know, Flink currently doesn't have a built-in throttling function. 
You can write your own user-defined function to achieve this. Your function 
just gives out what it reads in and limits the speed it gives out records at 
the same time.



If you're not familiar with user-defined functions, see 
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/udfs.html



Here is a throttling iterator example: 
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java




M Singh  于2019年11月25日周一 上午5:50写道：





Hi:



I have an Flink streaming application that invokes  some other web services.  
However the webservices have limited throughput.  So I wanted to find out if 
there any recommendations on how to throttle the Flink datastream so that they 
don't overload the downstrream services.  I am using Kinesis as source and sink 
in my application.



Please let me know if there any hooks available in Flink, what are the patterns 
that can be used and what are the best practices/pitfalls for using them.


Thanks 


Mans

Re: Apache Flink - Throttling stream flow

2019-11-25 Thread Caizhi Weng 
Hi,

As far as I know, Flink currently doesn't have a built-in throttling
function. You can write your own user-defined function to achieve this.
Your function just gives out what it reads in and limits the speed it gives
out records at the same time.

If you're not familiar with user-defined functions, see
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/udfs.html

Here is a throttling iterator example:
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java

M Singh  于2019年11月25日周一 上午5:50写道：

> Hi:
>
> I have an Flink streaming application that invokes  some other web
> services.  However the webservices have limited throughput.  So I wanted to
> find out if there any recommendations on how to throttle the Flink
> datastream so that they don't overload the downstrream services.  I am
> using Kinesis as source and sink in my application.
>
> Please let me know if there any hooks available in Flink, what are the
> patterns that can be used and what are the best practices/pitfalls for
> using them.
>
> Thanks
>
> Mans
>

Apache Flink - Throttling stream flow

2019-11-24 Thread M Singh 
Hi:
I have an Flink streaming application that invokes  some other web services.  
However the webservices have limited throughput.  So I wanted to find out if 
there any recommendations on how to throttle the Flink datastream so that they 
don't overload the downstrream services.  I am using Kinesis as source and sink 
in my application.
Please let me know if there any hooks available in Flink, what are the patterns 
that can be used and what are the best practices/pitfalls for using them.
Thanks 
Mans