cshuo commented on issue #19025: URL: https://github.com/apache/hudi/issues/19025#issuecomment-5406799459
> one question, why acquiring the data table lock for MDT table services, can the MDT hold the lock by it's own since the plan scheduling and timeline publish are all independent from the data table. For Compaction execution, yes, the data table lock is not needed. For compaction scheduling, however, still reads the DT timeline to derive a safe MDT compaction instant: [`compactIfNecessary()`](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java#L1625-L1647). The lock in [`scheduleTableServiceInternal()`](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java#L720-L783) only covers plan generation/publication, after that instant has already been computed. Therefore, using an independent MDT-only lock would allow the DT timeline to change between instant generation and plan publication. The initial implementation holds the DT lock around the complete scheduling phase and execution phase uses the OCC MDT writer from #18295 without an outer DT lock. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
