[
https://issues.apache.org/jira/browse/HUDI-7574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-7574:
---------------------------------
Status: Patch Available (was: In Progress)
> Auto-scaling for Flink Hudi sink tasks
> --------------------------------------
>
> Key: HUDI-7574
> URL: https://issues.apache.org/jira/browse/HUDI-7574
> Project: Apache Hudi
> Issue Type: Improvement
> Components: flink
> Reporter: Danny Chen
> Assignee: Danny Chen
> Priority: Major
> Fix For: 1.0.0
>
>
> Currently the flink write task parallelism is set up throughÂ
> {code:java}
> write.tasks{code}
> it is kind of a fixed number during the lifecycle of the ingestion pipeline,
> while for streaming, there are always fluctuation of the workload, it is
> great if we can tune the parallelism of write tasks based on the job load
> profile dynamically.
> On K8s, Flink provides a
> [https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/]
> which is suitable for the purpose, which deserves a further investigation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
