yihua opened a new pull request, #19239: URL: https://github.com/apache/hudi/pull/19239
### Describe the issue this Pull Request addresses With `hoodie.datasource.meta.sync.condition.sync=true`, `HiveSyncTool` only advances the last-commit-time-synced marker when the schema, table properties, or partitions changed. A table that keeps receiving commits with no metastore-visible changes (for example, no-op commits from a streaming pipeline on an idle source) never advances the marker. Once the marker falls behind the start of the active timeline, `HoodieSyncClient.getDroppedPartitionsSince` via `TimelineUtils.getCommitsTimelineAfter` merges the archived timeline and reads commit metadata per instant on every subsequent sync round, which takes minutes on tables with large archived timelines. ### Summary and Changelog When conditional sync would skip the marker update, advance the marker anyway if it trails the midpoint of the active commits timeline: - `HiveSyncTool.syncHoodieTable`: add `isLastCommitTimeSyncedBehindTimelineMidpoint(tableName)` to the marker-update condition. - The check compares the stored marker against the requested time of the middle instant of the completed commits timeline, so the marker always stays in the newer half of the active timeline and never ages into the archived timeline, while the catalog property write remains infrequent. - Updated `testSyncWithoutDiffs` to the new semantics: the marker advances to the latest commit once it trails the midpoint, both while it is still on the active timeline and after it has fallen behind the timeline start. ### Impact Meta sync with conditional sync enabled now refreshes the `last_commit_time_sync` and `last_commit_completion_time_sync` table properties occasionally on no-change syncs (only when the marker crosses the timeline midpoint). This bounds marker staleness and avoids the archived-timeline reads in written/dropped partition computation. No API changes. ### Risk Level low. The forced update reuses the existing `updateLastCommitTimeSynced` path, and behavior without conditional sync is unchanged. Verified with the updated `testSyncWithoutDiffs` across sync modes. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
