morningman opened a new pull request, #66510:
URL: https://github.com/apache/doris/pull/66510
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #66400
Problem Summary:
Part 2 of the BE build-speed series started in #66400. That PR cut the
high-payload include edges out of the `exec_env.h` / `runtime_state.h` /
`thread_context.h` superhighways; this one finishes the include-edge line and
then goes after a cost the include audit cannot see: **template-instantiation
amplifiers** — small headers whose inline bodies force hundreds of TUs to
instantiate expensive template chains they never use.
Measured with the `--compile-bench` harness from #66400 (cold, cache-free,
`-j5`, Apple M5 Pro, clang 20), same code base A/B:
| wave | build wall | delta |
|---|---|---|
| baseline (with #66400 merged) | 30m48s | |
| three more hot edges: pod_array / column.h / wide_integer-boost | 30m24s |
-1.3% wall, -3.6% be/src CPU |
| RLE + FMT_COMPILE instantiation amplifiers | 28m19s | -6.9%, 483 files
faster |
**This PR: 30m48s -> 28m19s (-8.1%). Cumulative with #66400: 38m54s -> 28m19s
build wall (-27%).**
Wave A — three more hot include edges (commits 1+2, prepare/cut):
- `core/pod_array.h -> runtime/thread_context.h`: dead include left over from
the PODArray memory-tracking experiment (#50549); dragged thread_context /
exec_env into 203 TUs of core/
- `core/column/column.h -> exec/sort/hybrid_sorter.h`: core->exec layering
violation reaching 808 TUs; HybridSorter is only named in virtual
signatures
(now forward-declared, the BE_TEST-only body sunk into column.cpp)
- `core/wide_integer_impl.h -> boost/multiprecision/cpp_bin_float.hpp`:
4.27 MB of preprocessed closure in 1169 TUs, needed only for the
from-double conversion on platforms without an 80-bit long double; now
compiled once in a dedicated TU (`core/wide_integer_from_double.cpp`) with
explicit instantiations, x86 semantics untouched. `column.h` closure:
16.72 MB -> 10.15 MB (-39%)
Wave B — instantiation amplifiers (commits 3+4, prepare/cut):
- Both parquet `decoder.h` headers defined `BaseDictDecoder`'s virtual
methods inline, and those bodies construct/call
`RleBatchDecoder<uint32_t>`.
A call inside an inline body implicitly instantiates the referenced
templates in **every TU that includes the header**, so 531 TUs (cloud/,
information_schema/, python UDF... most of them nowhere near parquet) each
spent ~435 ms instantiating the GetBatch -> GetLiteralValues -> UnpackBatch
-> UnpackValues chain: 231 CPU s total, plus the 2038-line
rle_encoding/bit_stream_utils/bit_packing family reparsed in each.
Everything moved out of line is a per-page/per-batch **virtual** call that
is already dispatched through the vtable at every call site, so outlining
changes no generated call. The real decode TUs keep including
rle_encoding.h directly and inline the chain exactly as before; the
per-value-hot `LevelDecoder::get_next` is deliberately untouched.
- `core/uint24.h::to_string` held the single biggest fmt instantiation in the
codebase (`FMT_COMPILE("{:04d}-{:02d}-{:02d}")`, 53.5 CPU s over 1147 TUs);
it and `LargeIntValue`'s int128 formatters moved into .cpp files **keeping
FMT_COMPILE** — the runtime formatting path is instruction-identical, only
the allocation-dominated `std::string` builders lose cross-TU inlining.
Why extern template was rejected for the RLE chain: the methods are defined
in-class, hence implicitly inline, and explicit instantiation declarations do
not suppress implicit instantiation of inline functions ([temp.explicit]);
moving them out-of-class to make it work would de-inline the hot decode loops
in the real decode TUs. Outlining the cold virtual callers is strictly
better.
Guards: 4 new rules in `build-support/check-header-deps.py` (19 total, all
passing) pin `pod_array.h`, `column.h` and both `decoder.h` headers away from
the cut subtrees; the fmt cuts are recorded as a comment (the scanner only
follows quoted project includes).
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- `build-support/compile-bench/syntax_sweep.py`: 1364/1364 be/src TUs
pass `-fsyntax-only` after every cut (re-run on the rebased tree
against current master)
- `build-support/check-header-deps.py`: 19/19 layering rules pass
- full `sh build.sh --compile-bench` cold builds succeed after each
wave; `report.py compare` shows 483 files faster / regressions
confined to the scheduling-sensitive 4 GB tail TUs whose include
closure is untouched by this PR (machine noise, same signature as
documented in #66400)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
