sivabalan narayanan created HUDI-9067:
-----------------------------------------
Summary: Clean action sometimes falls back to
spark.default.parallelism set in the env
Key: HUDI-9067
URL: https://issues.apache.org/jira/browse/HUDI-9067
Project: Apache Hudi
Issue Type: Improvement
Components: cleaning
Reporter: sivabalan narayanan
Rarely, we notice there are 1024 (spark.default.parallelism) tasks spinning up
for clean actions.
for eg, if we try to ingest very small no of records say just 1, clean action
executor spins up 1024 tasks even though the clean parallelism is set to small
value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
