Re: [DISCUSS FLINKSQL PARALLELISM]

[liu ron]

Hi, Green

Thanks for driving this discussion, in batch mode we have the Adaptive
Batch Scheduler which automatically derives operator parallelism based on
data volume at runtime, so we don't need to care about the parallelism.
However, in stream mode, currently, Flink SQL can only set the parallelism
of an operator globally, and many users would like to set the parallelism
of an operator individually, which seems to be a pain point at the moment,
and it would make sense to support set parallelism at operator granularity.
Do you have any idea about the solution for this problem?

Best,
Ron


GREEN <1286649...@qq.com.invalid> 于2023年4月14日周五 16:03写道：

> Problem：&nbsp;
>
>
> Currently, FlinkSQL can&nbsp; set a unified parallelism in the job,it
> cannot set parallelism for each operator.
> This can cause resource waste&nbsp; On the occasion of&nbsp; high
> parallelism and small data volume.there may also be too many small
> file&nbsp; for&nbsp; writing HDFS Scene.
>
>
> Solution：
> I can modify FlinkSQL to support operator parallelism.Is it meaningful to
> do this？Let's discuss.