hudi-bot opened a new issue, #16439:
URL: https://github.com/apache/hudi/issues/16439
Currently the flink write task parallelism is set up through
{code:java}
write.tasks{code}
it is kind of a fixed number during the lifecycle of the ingestion pipeline,
while for streaming, there are always fluctuation of the workload, it is great
if we can tune the parallelism of write tasks based on the job load profile
dynamically.
On K8s, Flink provides a
[https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/]
which is suitable for the purpose, which deserves a further investigation.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7574
- Type: Improvement
- Fix version(s):
- 1.2.0
---
## Comments
09/Apr/24 01:45;vinoth;We need to rethink these singleton tasks like
cleaning etc. ;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
