liaoxin01 opened a new pull request, #67972:
URL: https://github.com/apache/doris/pull/67972
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
`CloudTabletMgr::sync_tablets()` selected the tablets it would work on by
`last_sync_time_s`, then did two things to each one: `sync_meta()` followed by
`sync_rowsets()`.
But `last_sync_time_s` only tracks how stale a tablet's **rowsets** are. It
is advanced inside `sync_tablet_rowsets()`, and only when that actually issues
its RPC. A tablet under continuous ingest therefore keeps it permanently fresh,
never falls below the staleness bound, and never has `sync_meta()` called on it
at all.
Its `TabletMeta` then stays at whatever it was built with for the lifetime
of the object, so properties that only arrive through the tablet meta are never
picked up. `ttl_seconds` is the one that shows: `CloudTablet::sync_meta()` is
the only thing that refreshes it, and it feeds the file cache expiration
computed on the write path (`CloudRowsetBuilder`, compaction output), the read
path (`TabletReader`, `OlapScanner`) and warm-up. An `ALTER TABLE ... SET
("file_cache_ttl_seconds" = ...)` on a table under load therefore has no effect
on those blocks, which keep landing in the wrong queue with an expiration
derived from the stale value.
### Release note
Fixed tablet metadata never being refreshed for tablets under continuous
ingest, which left properties such as `file_cache_ttl_seconds` stale on those
tablets after an ALTER.
### What is changed and how it works?
Give meta staleness its own clock. `last_sync_tablet_meta_time_s` is
advanced only by `sync_meta()`, next to `last_sync_time_s` which stays as it is
and keeps tracking rowsets.
`sync_tablets()` now decides per tablet which of the two RPCs it is due for,
rather than sorting tablets into a single bucket:
- rowsets stale -> sync rowsets, and the meta too. Pulling rowsets implies
pulling the meta, which is the relationship the single pass had: the rowsets
are only as trustworthy as the meta they belong to.
- rowsets fresh but meta stale -> sync the meta only, one RPC instead of the
two a full sync costs. This is the case that used to be skipped entirely.
- both fresh -> skip.
Work is still ordered by the older of the two clocks, so a mid-run stop has
already served the tablets that waited longest.
New bvars `sync_tablets_meta_num` and `sync_tablets_rowsets_num` split what
`num_sync` used to lump together, and the finish log reports both.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
`CloudTabletMgrTest.SyncTabletsRefreshesMetaOfContinuouslyIngestedTablet`
covers all three cases: a tablet whose rowset clock is fresh while its meta
clock is stale is synced for meta only, one stale in both is synced for both,
one fresh in both is skipped.
```bash
DORIS_TOOLCHAIN=clang DISABLE_BE_JAVA_EXTENSIONS=ON
ENABLE_INJECTION_POINT=ON ENABLE_PCH=0 sh run-be-ut.sh --run
--filter='CloudTabletMgrTest.*'
```
- [ ] This is a refactor/code format and no logic has been changed.
- Behavior changed:
- [ ] No.
- [x] Yes. Tablets whose rowsets are synced continuously now get their
tablet meta refreshed on the normal staleness interval. Tablets that need meta
only no longer pay for a rowset sync as well.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
