[ https://issues.apache.org/jira/browse/HUDI-7574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835092#comment-17835092 ]
Vinoth Chandar commented on HUDI-7574: -------------------------------------- We need to rethink these singleton tasks like cleaning etc. > Auto-pilot for Flink Hudi sink tasks > ------------------------------------ > > Key: HUDI-7574 > URL: https://issues.apache.org/jira/browse/HUDI-7574 > Project: Apache Hudi > Issue Type: Improvement > Components: flink > Reporter: Danny Chen > Assignee: Danny Chen > Priority: Major > Fix For: 1.0.0 > > > Currently the flink write task parallelism is set up through > {code:java} > write.tasks{code} > it is kind of a fixed number during the lifecycle of the ingestion pipeline, > while for streaming, there are always fluctuation of the workload, it is > great if we can tune the parallelism of write tasks based on the job load > profile dynamically. > On K8s, Flink provides a > [https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/] > which is suitable for the purpose, which deserves a further investigation. -- This message was sent by Atlassian Jira (v8.20.10#820010)