yihua opened a new pull request #3991: URL: https://github.com/apache/hudi/pull/3991
## What is the purpose of the pull request This PR adds an option to run async compaction and clustering jobs without providing the instant time, using the earliest instant of the corresponding requested action by default, to improve usability. ## Brief change log - Adds an API in `HoodieTimeline` to find the first instant of a specific action and state. - For `HoodieClusteringJob`, when `execute` mode is used, if the instant time is not provided from `--instant-time`, the job automatically finds the earliest scheduled clustering instant for execution. - For `HoodieCompactor`, when executing compaction, if the instant time is not provided from `--instant-time`, the job automatically finds the earliest scheduled compaction instant for execution. ## Verify this pull request This change added tests in `TestHoodieDeltaStreamer#testHoodieAsyncClusteringJob`. The change has also been verified by running `HoodieCompactor` and `HoodieClusteringJob` with kafka-connect: - Compaction scheduled, run `HoodieCompactor`, earliest requested compaction instant is executed. - Compaction done, run `HoodieCompactor`, throws an exception. - Clustering scheduled, run `HoodieClusteringJob`, earliest requested clustering instant is executed. - Clustering done, run `HoodieClusteringJob`, throws an exception. Async compaction command: ``` spark-submit \ --class org.apache.hudi.utilities.HoodieCompactor \ hudi/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.10.0-SNAPSHOT.jar \ --base-path /tmp/hoodie/hudi-test-topic \ --table-name hudi-test-topic \ --schema-file /repo/hudi/docker/demo/config/schema.avsc \ --parallelism 2 \ --spark-memory 1g ``` Async clustering command: ``` spark-submit \ --class org.apache.hudi.utilities.HoodieClusteringJob \ hudi/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.10.0-SNAPSHOT.jar \ --props clusteringjob.properties \ --mode execute \ --base-path /tmp/hoodie/hudi-test-topic \ --table-name sample_table \ --spark-memory 1g ``` ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org