[
https://issues.apache.org/jira/browse/FLINK-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
swy updated FLINK-9597:
-----------------------
Description:
Hi, we found that our Flink application with simple logic, which using process
function is not scale-able when scale from 8 parallelism onward even though
with sufficient resources. Below it the result which is capped at ~250k TPS. No
matter how we tune the parallelism of the operators it just not scale, same to
increase source parallelism.
Please refer to "scaleNotWork.png",
1. fixed source parallelism 4, other operators parallelism 8
2. fixed source parallelism 4, other operators parallelism 16
3. fixed source parallelism 4, other operators parallelism 32
4. fixed source parallelism 6, other operators parallelism 8
5. fixed source parallelism 6, other operators parallelism 16
6. fixed source parallelism 6, other operators parallelism 32
7. fixed source parallelism 6, other operators parallelism 64 performance worse
than parallelism 32.
Sample source code attached(flink_app_parser_git.zip). It is a simple program,
parsing json record into object, and pass it to a empty logic Flink's process
function. Rocksdb is in used, and the source is generated by the program
itself. This could be reproduce easily.
We choose Flink because of it scalability, but this is not the case now,
appreciated if anyone could help as this is impacting our projects! thank you.
To run the program, sample parameters,
"aggrinterval=6000000 loop=7500000 statsd=1 psrc=4 pJ2R=32 pAggr=72
URL=do36.comptel.com:8127"
* aggrinterval: time in ms for timer to trigger
* loop: how many row of data to feed
* statsd: to send result to statsd
* psrc: source parallelism
* pJ2R: parallelism of map operator(JsonRecTranslator)
* pAggr: parallelism of process+timer operator(AggregationDuration) !JM.png!
!TM.png!
was:
Hi, we found that our Flink application with simple logic, which using process
function is not scale-able when scale from 8 parallelism onward even though
with sufficient resources. Below it the result which is capped at ~250k TPS. No
matter how we tune the parallelism of the operators it just not scale, same to
increase source parallelism.
Please refer to "scaleNotWork.png",
1. fixed source parallelism 4, other operators parallelism 8
2. fixed source parallelism 4, other operators parallelism 16
3. fixed source parallelism 4, other operators parallelism 32
4. fixed source parallelism 6, other operators parallelism 8
5. fixed source parallelism 6, other operators parallelism 16
6. fixed source parallelism 6, other operators parallelism 32
7. fixed source parallelism 6, other operators parallelism 64 performance worse
than parallelism 32.
Sample source code attached(flink_app_parser_git.zip). It is a simple program,
parsing json record into object, and pass it to a empty logic Flink's process
function. Rocksdb is in used, and the source is generated by the program
itself. This could be reproduce easily.
We choose Flink because of it scalability, but this is not the case now,
appreciated if anyone could help as this is impacting our projects! thank you.
To run the program, sample parameters,
"aggrinterval=6000000 loop=7500000 statsd=1 psrc=4 pJ2R=32 pAggr=72
URL=do36.comptel.com:8127"
* aggrinterval: time in ms for timer to trigger
* loop: how many row of data to feed
* statsd: to send result to statsd
* psrc: source parallelism
* pJ2R: parallelism of map operator(JsonRecTranslator)
* pAggr: parallelism of process+timer operator(AggregationDuration)
> Flink fail to scale!
> --------------------
>
> Key: FLINK-9597
> URL: https://issues.apache.org/jira/browse/FLINK-9597
> Project: Flink
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.5.0
> Reporter: swy
> Priority: Major
> Attachments: flink_app_parser_git.zip, scaleNotWork.png
>
>
> Hi, we found that our Flink application with simple logic, which using
> process function is not scale-able when scale from 8 parallelism onward even
> though with sufficient resources. Below it the result which is capped at
> ~250k TPS. No matter how we tune the parallelism of the operators it just not
> scale, same to increase source parallelism.
> Please refer to "scaleNotWork.png",
> 1. fixed source parallelism 4, other operators parallelism 8
> 2. fixed source parallelism 4, other operators parallelism 16
> 3. fixed source parallelism 4, other operators parallelism 32
> 4. fixed source parallelism 6, other operators parallelism 8
> 5. fixed source parallelism 6, other operators parallelism 16
> 6. fixed source parallelism 6, other operators parallelism 32
> 7. fixed source parallelism 6, other operators parallelism 64 performance
> worse than parallelism 32.
> Sample source code attached(flink_app_parser_git.zip). It is a simple
> program, parsing json record into object, and pass it to a empty logic
> Flink's process function. Rocksdb is in used, and the source is generated by
> the program itself. This could be reproduce easily.
> We choose Flink because of it scalability, but this is not the case now,
> appreciated if anyone could help as this is impacting our projects! thank you.
> To run the program, sample parameters,
> "aggrinterval=6000000 loop=7500000 statsd=1 psrc=4 pJ2R=32 pAggr=72
> URL=do36.comptel.com:8127"
> * aggrinterval: time in ms for timer to trigger
> * loop: how many row of data to feed
> * statsd: to send result to statsd
> * psrc: source parallelism
> * pJ2R: parallelism of map operator(JsonRecTranslator)
> * pAggr: parallelism of process+timer operator(AggregationDuration) !JM.png!
> !TM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
