[
https://issues.apache.org/jira/browse/FLINK-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ryantaocer updated FLINK-12002:
-------------------------------
Description:
In Flink the parallelism of job is a pre-specified parameter, which is usually
an empirical value and thus might not be optimal for both performance and
resource depending on the amount of data processed in each task.
Furthermore, a fixed parallelism cannot scale to varying data size common in
production cluster where we may not often change configurations.
We propose to determine the job parallelism adaptive to the actual total input
data size and an ideal data size processed by each task. The ideal size is
pre-specified according to the properties of the operator such as the
preparation overhead compared with data processing time.
detailed design doc coming soon.
was:
In Flink the parallelism of job is a pre-specified parameter, which is usually
an empirical value and thus might not be optimal for both performance and
resource depending on the amount of data processed in each task.
Furthermore, a fixed parallelism cannot scale to varying data size common in
production cluster where we may not often change configurations.
We propose to determine the job parallelism adaptive to the actual total input
data size and an ideal data size processed by each task. The ideal size is
pre-specified according to the properties of the operator such as the
preparation overhead compared with data processing time.
detailed design doc coming soon.
> Adaptive Parallelism of Job Vertex Execution
> --------------------------------------------
>
> Key: FLINK-12002
> URL: https://issues.apache.org/jira/browse/FLINK-12002
> Project: Flink
> Issue Type: Improvement
> Reporter: ryantaocer
> Assignee: BoWang
> Priority: Major
>
> In Flink the parallelism of job is a pre-specified parameter, which is
> usually an empirical value and thus might not be optimal for both performance
> and resource depending on the amount of data processed in each task.
> Furthermore, a fixed parallelism cannot scale to varying data size common in
> production cluster where we may not often change configurations.
> We propose to determine the job parallelism adaptive to the actual total
> input data size and an ideal data size processed by each task. The ideal size
> is pre-specified according to the properties of the operator such as the
> preparation overhead compared with data processing time.
> detailed design doc coming soon.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
