Re: how to simulate the scene "back pressure" in flink?Thanks~!

2020-10-10 Thread David Anderson 
>
> Could I use your command with no docker?


Hypothetically, yes, but it's a somewhat impractical idea. The
ClickCountJob needs Flink and Kafka, and there is another java application
(the clickevent-generator) that writes into Kafka the data that is being
processed.

On Sat, Oct 10, 2020 at 5:32 PM 大森林  wrote:

>
> Could I use your command with no docker?
>
> -- 原始邮件 --
> *发件人:* "David Anderson" ;
> *发送时间:* 2020年10月10日(星期六) 晚上10:30
> *收件人:* "大森林";
> *抄送:* "Arvid Heise";"user";
> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>
> The ClickCountJob used in the operations playground accepts an application
> parameter, like this:
>
> flink run -d /opt/ClickCountJob.jar --bootstrap.servers kafka:9092
> --checkpointing --event-time --backpressure
>
> To try this, you would modify the docker-compose.yaml file in [1]. If you
> want to see how it is implemented, see [2].
>
> You can not use this --backpressure option with any other application.
>
> [1]
> https://github.com/apache/flink-playgrounds/blob/release-1.11/operations-playground/docker-compose.yaml
> [2]
> https://github.com/apache/flink-playgrounds/blob/release-1.11/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/BackpressureMap.java
>
>
> On Sat, Oct 10, 2020 at 7:26 AM 大森林  wrote:
>
>> Thanks for both of your help...
>> but...
>>
>> I can not understand both:
>>
>> 
>> Dear David Anderson:
>> Is the whole command like this?
>> flink run *--backpressure*  -c wordcount_increstate
>> datastream_api-1.0-SNAPSHOT.jar
>>
>>
>> 
>> Dear Arvid Heise:
>> For conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
>> will this settings work to sleep when the output stream is generating?
>>
>>
>> 
>> apologise for my poor basic knowledge of flink~
>> Thanks for both of your help~
>>
>>
>>
>> -- 原始邮件 --
>> *发件人:* "David Anderson" ;
>> *发送时间:* 2020年10月9日(星期五) 晚上8:23
>> *收件人:* "Arvid Heise";
>> *抄送:* "大森林";"user";
>> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>>
>> The Flink Operations Playground includes an optional backpressure
>> simulation you can experiment with. It is described at the end of [1].
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/flink-operations-playground.html#variants
>>
>>
>> On Fri, Oct 9, 2020 at 10:02 AM Arvid Heise  wrote:
>>
>>> You can add
>>>
>>> conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
>>>
>>> at any place before creating the environment [1]. Default value is 8081,
>>> so you can access web ui through http://localhost:8081, but you can
>>> really choose any other free port.
>>>
>>> [1]
>>> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L198
>>> On Fri, Oct 9, 2020 at 9:24 AM 大森林  wrote:
>>>
  Thanks for your repies,
 could you tell me where to set RestOption.POPT?in configuration
 what's the value should I set for RestOption.PORT?

 Thanks.


 -- 原始邮件 --
 *发件人:* "Arvid Heise" ;
 *发送时间:* 2020年10月9日(星期五) 下午3:00
 *收件人:* "大森林";
 *抄送:* "David,Anderson";"user"<
 user@flink.apache.org>;
 *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!

 The easiest way to see backpressure is to add some sleep to your sink,
 check [1] for an example.
 If you execute that unit test with a RestOption.PORT set in the
 configuration, you can even load the Web UI and watch the backpressure
 accumulate and finally go away at the end of the test.

 [1]
 https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536

 On Tue, Oct 6, 2020 at 4:13 PM 大森林  wrote:

>
> I want to learn the concept "back pressure".
> but I can not find the datastream generator example to generate a lot
> of data.
>
> besides,
> is there any example on how to simulate the scene "back pressure"  in
> WEB UI?
>
> Thanks for your help~
>


 --

 Arvid Heise | Senior Java Developer

 

 Follow us @VervericaData

 --

 Join Flink Forward  - The Apache Flink
 Conference

 Stream Processing | Event Driven | Real Time

 --

 Ververica GmbH | Invalidenstrasse 115, 10115 Berlin,

Re: how to simulate the scene "back pressure" in flink?Thanks~!

2020-10-10 Thread David Anderson 
The ClickCountJob used in the operations playground accepts an application
parameter, like this:

flink run -d /opt/ClickCountJob.jar --bootstrap.servers kafka:9092
--checkpointing --event-time --backpressure

To try this, you would modify the docker-compose.yaml file in [1]. If you
want to see how it is implemented, see [2].

You can not use this --backpressure option with any other application.

[1]
https://github.com/apache/flink-playgrounds/blob/release-1.11/operations-playground/docker-compose.yaml
[2]
https://github.com/apache/flink-playgrounds/blob/release-1.11/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/BackpressureMap.java


On Sat, Oct 10, 2020 at 7:26 AM 大森林  wrote:

> Thanks for both of your help...
> but...
>
> I can not understand both:
>
> 
> Dear David Anderson:
> Is the whole command like this?
> flink run *--backpressure*  -c wordcount_increstate
> datastream_api-1.0-SNAPSHOT.jar
>
>
> 
> Dear Arvid Heise:
> For conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
> will this settings work to sleep when the output stream is generating?
>
>
> 
> apologise for my poor basic knowledge of flink~
> Thanks for both of your help~
>
>
>
> -- 原始邮件 --
> *发件人:* "David Anderson" ;
> *发送时间:* 2020年10月9日(星期五) 晚上8:23
> *收件人:* "Arvid Heise";
> *抄送:* "大森林";"user";
> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>
> The Flink Operations Playground includes an optional backpressure
> simulation you can experiment with. It is described at the end of [1].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/flink-operations-playground.html#variants
>
>
> On Fri, Oct 9, 2020 at 10:02 AM Arvid Heise  wrote:
>
>> You can add
>>
>> conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
>>
>> at any place before creating the environment [1]. Default value is 8081,
>> so you can access web ui through http://localhost:8081, but you can
>> really choose any other free port.
>>
>> [1]
>> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L198
>> On Fri, Oct 9, 2020 at 9:24 AM 大森林  wrote:
>>
>>>  Thanks for your repies,
>>> could you tell me where to set RestOption.POPT?in configuration
>>> what's the value should I set for RestOption.PORT?
>>>
>>> Thanks.
>>>
>>>
>>> -- 原始邮件 --
>>> *发件人:* "Arvid Heise" ;
>>> *发送时间:* 2020年10月9日(星期五) 下午3:00
>>> *收件人:* "大森林";
>>> *抄送:* "David,Anderson";"user"<
>>> user@flink.apache.org>;
>>> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>>>
>>> The easiest way to see backpressure is to add some sleep to your sink,
>>> check [1] for an example.
>>> If you execute that unit test with a RestOption.PORT set in the
>>> configuration, you can even load the Web UI and watch the backpressure
>>> accumulate and finally go away at the end of the test.
>>>
>>> [1]
>>> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
>>>
>>> On Tue, Oct 6, 2020 at 4:13 PM 大森林  wrote:
>>>

 I want to learn the concept "back pressure".
 but I can not find the datastream generator example to generate a lot
 of data.

 besides,
 is there any example on how to simulate the scene "back pressure"  in
 WEB UI?

 Thanks for your help~

>>>
>>>
>>> --
>>>
>>> Arvid Heise | Senior Java Developer
>>>
>>> 
>>>
>>> Follow us @VervericaData
>>>
>>> --
>>>
>>> Join Flink Forward  - The Apache Flink
>>> Conference
>>>
>>> Stream Processing | Event Driven | Real Time
>>>
>>> --
>>>
>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>
>>> --
>>> Ververica GmbH
>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>>> (Toni) Cheng
>>>
>>
>>
>> --
>>
>> Arvid Heise | Senior Java Developer
>>
>> 
>>
>> Follow us @VervericaData
>>
>> --
>>
>> Join Flink Forward  - The Apache Flink
>> Conference
>>
>> Stream Processing | Event Driven | Real Time
>>
>> --
>>
>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>
>> --
>> Ververica GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>> (Toni) Cheng
>>
>

Re: how to simulate the scene "back pressure" in flink?Thanks~!

2020-10-09 Thread David Anderson 
The Flink Operations Playground includes an optional backpressure
simulation you can experiment with. It is described at the end of [1].

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/flink-operations-playground.html#variants


On Fri, Oct 9, 2020 at 10:02 AM Arvid Heise  wrote:

> You can add
>
> conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
>
> at any place before creating the environment [1]. Default value is 8081,
> so you can access web ui through http://localhost:8081, but you can
> really choose any other free port.
>
> [1]
> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L198
> On Fri, Oct 9, 2020 at 9:24 AM 大森林  wrote:
>
>>  Thanks for your repies,
>> could you tell me where to set RestOption.POPT?in configuration
>> what's the value should I set for RestOption.PORT?
>>
>> Thanks.
>>
>>
>> -- 原始邮件 --
>> *发件人:* "Arvid Heise" ;
>> *发送时间:* 2020年10月9日(星期五) 下午3:00
>> *收件人:* "大森林";
>> *抄送:* "David,Anderson";"user"<
>> user@flink.apache.org>;
>> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>>
>> The easiest way to see backpressure is to add some sleep to your sink,
>> check [1] for an example.
>> If you execute that unit test with a RestOption.PORT set in the
>> configuration, you can even load the Web UI and watch the backpressure
>> accumulate and finally go away at the end of the test.
>>
>> [1]
>> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
>>
>> On Tue, Oct 6, 2020 at 4:13 PM 大森林  wrote:
>>
>>>
>>> I want to learn the concept "back pressure".
>>> but I can not find the datastream generator example to generate a lot of
>>> data.
>>>
>>> besides,
>>> is there any example on how to simulate the scene "back pressure"  in
>>> WEB UI?
>>>
>>> Thanks for your help~
>>>
>>
>>
>> --
>>
>> Arvid Heise | Senior Java Developer
>>
>> 
>>
>> Follow us @VervericaData
>>
>> --
>>
>> Join Flink Forward  - The Apache Flink
>> Conference
>>
>> Stream Processing | Event Driven | Real Time
>>
>> --
>>
>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>
>> --
>> Ververica GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>> (Toni) Cheng
>>
>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> 
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward  - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>

Re: how to simulate the scene "back pressure" in flink?Thanks~!

2020-10-09 Thread Arvid Heise 
You can add

conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());

at any place before creating the environment [1]. Default value is 8081, so
you can access web ui through http://localhost:8081, but you can really
choose any other free port.

[1]
https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L198
On Fri, Oct 9, 2020 at 9:24 AM 大森林  wrote:

>  Thanks for your repies,
> could you tell me where to set RestOption.POPT?in configuration
> what's the value should I set for RestOption.PORT?
>
> Thanks.
>
>
> -- 原始邮件 --
> *发件人:* "Arvid Heise" ;
> *发送时间:* 2020年10月9日(星期五) 下午3:00
> *收件人:* "大森林";
> *抄送:* "David,Anderson";"user"<
> user@flink.apache.org>;
> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>
> The easiest way to see backpressure is to add some sleep to your sink,
> check [1] for an example.
> If you execute that unit test with a RestOption.PORT set in the
> configuration, you can even load the Web UI and watch the backpressure
> accumulate and finally go away at the end of the test.
>
> [1]
> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
>
> On Tue, Oct 6, 2020 at 4:13 PM 大森林  wrote:
>
>>
>> I want to learn the concept "back pressure".
>> but I can not find the datastream generator example to generate a lot of
>> data.
>>
>> besides,
>> is there any example on how to simulate the scene "back pressure"  in WEB
>> UI?
>>
>> Thanks for your help~
>>
>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> 
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward  - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>


-- 

Arvid Heise | Senior Java Developer



Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng

Re: how to simulate the scene "back pressure" in flink?Thanks~!

2020-10-09 Thread Arvid Heise 
The easiest way to see backpressure is to add some sleep to your sink,
check [1] for an example.
If you execute that unit test with a RestOption.PORT set in the
configuration, you can even load the Web UI and watch the backpressure
accumulate and finally go away at the end of the test.

[1]
https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536

On Tue, Oct 6, 2020 at 4:13 PM 大森林  wrote:

>
> I want to learn the concept "back pressure".
> but I can not find the datastream generator example to generate a lot of
> data.
>
> besides,
> is there any example on how to simulate the scene "back pressure"  in WEB
> UI?
>
> Thanks for your help~
>


-- 

Arvid Heise | Senior Java Developer



Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng