morningman opened a new pull request, #68797:
URL: https://github.com/apache/doris/pull/68797
### What problem does this PR solve?
Issue Number: None
Related PR: #66684
Original author: @hubgeter (the change is ported from #66684 on branch-4.1;
the commit carries Co-authored-by)
Problem Summary:
**In short.** branch-4.1 (from 4.1.5) and branch-4.2 enable the block file
cache by default. master still has it off. This difference matters most on
upgrade, through the BE config: BE configs are not persisted. A
storage-compute-coupled cluster that relies on the default therefore loses its
file cache when it moves from 4.1.5 / 4.2 to a release built from master. This
PR ports #66684 and makes both the BE config `enable_file_cache` and the
session variable `enable_file_cache` default to true.
**Background**
- Two switches gate the block file cache:
- The BE config `enable_file_cache` decides whether a BE creates the cache
at all. In cloud mode, BE forces it on after reading its config file and
refuses to start without it.
- The session variable `enable_file_cache` decides whether external file
scans read through the cache. `FileFactory::get_reader_options` requires the BE
config, the session variable and file cache admission to all be true. The
session variable has no effect on internal tables.
- With the session variable on, FE also assigns external scan ranges by
consistent hashing instead of round robin (`ExternalScanNode`). A file then
keeps landing on the same backends, so its cached blocks get reused.
- Session variables and BE configs behave differently across upgrades:
- FE writes every session variable value into its image
(`VariableMgr.write` → `SessionVariable.toJson`). An upgraded cluster keeps the
values it had whatever the new default is; only a new cluster gets a new
default.
- BE configs are not persisted. A BE uses the default compiled into its
binary unless be.conf sets the key.
**The problem, and what it cost**
| Default | master | branch-4.2 | 4.1.4 | 4.1.5 |
|---|---|---|---|---|
| BE config `enable_file_cache` | false | true | false | true |
| session variable `enable_file_cache` | false | true | false | true |
- Take a storage-compute-coupled cluster whose be.conf does not set
`enable_file_cache`. When it is upgraded from 4.1.5 / 4.2 to a release built
from master, it silently loses its file cache. External table scans and reads
of rowsets cooled down to remote storage go back to remote storage on every
query. The `${DORIS_HOME}/file_cache` directory is left on disk.
- New clusters built from master behave differently from new 4.1.5 / 4.2
clusters. In coupled mode they have no file cache at all, and in either mode
external scans are not cached.
- Cloud clusters are not affected by the BE default, because cloud mode
forces it on. Upgraded clusters keep their session variable value either way.
**How this PR fixes it**
As in #66684, both defaults become true: `DEFINE_Bool(enable_file_cache,
"true")` in `be/src/common/config.cpp` and `enableFileCache = true` in
`SessionVariable`. Nothing else changes.
On a storage-compute-coupled BE that does not configure the cache, this
turns on the following:
- The cache lives under `${DORIS_HOME}/file_cache` (the default
`file_cache_path`). Without `total_size` it may use the whole file system.
Eviction starts once the file system is 88% full and stops at 85%
(`file_cache_enter/exit_need_evict_cache_in_advance_percent`).
`storage_root_path` defaults to the same disk, whose flood stage is 90%, so a
busy cache keeps a shared disk at 85%–88%. To bound it, set `file_cache_path`
with a `total_size`; to keep the old behavior, set `enable_file_cache = false`
in be.conf.
- Only reads through a remote file system go through the cache: external
file scans and cooled-down rowsets. Local rowsets are read exactly as before.
**Results**
| Scenario | Before | After |
|---|---|---|
| New coupled cluster, external table scan | not cached, scan ranges by
round robin | cached, scan ranges by consistent hashing |
| New cloud cluster, external table scan | not cached | cached |
| Coupled cluster upgraded from 4.1.5 / 4.2, be.conf without
`enable_file_cache` | file cache turned off by the upgrade | file cache stays
on |
| Coupled cluster upgraded from 4.1.4 or older | unchanged | BE cache on
(cooled-down rowsets cached). The session variable keeps its persisted `false`,
so external scans stay uncached until `SET GLOBAL enable_file_cache = true` |
| Internal tables on local disks | unchanged | unchanged |
### Release note
The block file cache is now enabled by default: the BE config
`enable_file_cache` and the session variable `enable_file_cache` default to
true.
- On a storage-compute-coupled BE that does not set `file_cache_path`, the
cache is created under `${DORIS_HOME}/file_cache` and may grow until the disk
is 85%–88% full. To bound it, set `file_cache_path` with a `total_size`; to
keep the old behavior, set `enable_file_cache = false` in be.conf.
- Clusters upgraded from older versions keep their persisted session
variable value. To read external tables through the cache, run `SET GLOBAL
enable_file_cache = true`.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
FE UT: `bash run-fe-ut.sh --run
org.apache.doris.qe.SessionVariablesTest,org.apache.doris.qe.VariableMgrTest`
ran 32 tests with 0 failures.
BE UT and the regression pipelines are left to CI. On branch-4.1, #66684
passed BE UT, FE UT, P0, External, NonConcurrent, cloud_p0 and vault_p0.
External Regression already runs its BE with the cache on, because the pipeline
appends it to be.conf. P0 and NonConcurrent run their BEs with it off today and
will run with it on after this PR.
- Behavior changed:
- [ ] No.
- [x] Yes. Both defaults change from false to true; see Results.
- Does this need documentation?
- [ ] No.
- [x] Yes. The documented defaults of the BE config and the session
variable change; a doris-website PR will follow.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
