Joy-2000 opened a new issue, #19714: URL: https://github.com/apache/hudi/issues/19714
### Context Split out from the broader lock-scope discussion in #19587, to keep the design and testing focused. ### Problem `scheduleTableServiceInternal` performs the entire scheduling process for clustering, compaction, and log compaction inside the transaction, including several expensive operations: - Loading the timeline - Building the filesystem view - Building the table-service plan The execution time of these operations varies significantly under filesystem load, so they can dominate lock-hold time. ### Proposal Move plan building and other expensive scheduling work outside the lock, keeping only the operations that require serialization (e.g. `createNewInstantTime`, `createTable`，`plan-file creation`) inside it. ### Potential risk Without fencing, concurrent table-service jobs may schedule overlapping work. If not detected or rejected, this can lead to duplicate processing or correctness issues. A **two-phase design** therefore seems necessary: 1. Outside the lock: load the timeline, build the filesystem view, build the plan. 2. Under the transaction lock (validation/fencing): reload the timeline/view, verify the plan's input file groups are still eligible and do not overlap any newly pending service, and reject/rebuild on failure before publishing. ### Test plan A concurrency test where two planners pause after selecting the same file groups, then publish in sequence — asserting the second publish is rejected/rebuilt rather than committing overlapping work. ### Expected benefit Reduce the duration the transaction lock is held during table-service scheduling, while preserving the current no-overlap guarantee via the fencing step. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
