This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b9ee837a3e6 [opt](build) Enable unity builds for the remaining BE
targets (#66789)
b9ee837a3e6 is described below
commit b9ee837a3e691e6354dfdc8e1a550d8fca4d3409
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sun Aug 16 12:48:31 2026 +0800
[opt](build) Enable unity builds for the remaining BE targets (#66789)
> Part of the BE build-time optimization series tracked in #66715.
>
> Split out of **https://github.com/apache/doris/pull/66510**, which
carries the whole
> BE build-time batch. #66712 introduced the `ENABLE_UNITY_BUILD` switch
and piloted
> unity builds on three low-risk glue targets; #66776 extended it to
Exec and Exprs.
> This PR finishes the line: unity builds for every remaining BE target
where it
> helps, plus a fail-loud guard for the skip lists. It is the last PR of
the unity
> sub-series (the remaining two PRs of the batch are orthogonal:
extern-template
> convergence and olap_common decoupling).
### What problem does this PR solve?
Related PR: #66510, #66712, #66776
Problem Summary:
Same mechanism as #66712/#66776: most of the cold-build cost of
glue-heavy targets
is **re-parsing the shared header closure once per small `.cpp`**, and
CMake's
`UNITY_BUILD` makes a batch pay that parse once. This PR applies it to
the twelve
remaining targets where the win is real, and writes down (in-tree) the
two places
where it is not.
The eleven commits, in fix-then-enable pairs:
1. **Deduplicate cloud/storage file-scope names that clash under unity**
—
identical copies hoisted to one shared definition (warm-up metric window
spans -> `bvar_windowed_adder.h`; the "D0R1" segment footer magic ->
`segment/common.h`; the `path_allocated_bytes` PathInData accounting
helper
that #66204 landed in two variant v2 TUs -> `util/json/path_in_data.h`),
a
divergent copy renamed instead (`create_ext_meta_io_context` keeps
file-cache stats where its siblings reset them); **plus one real latent
bug**: `merger.cpp` threw with `INTERNAL_ERROR` bound to
`PPlanFragmentCancelReason::INTERNAL_ERROR` (=3) from `types.pb.h`
instead
of `ErrorCode::INTERNAL_ERROR` (=6) — no `ErrorCode` using-directive was
in
scope. Qualified; those compaction-failure statuses now carry the
intended
code.
2. **Unity for Cloud (batch 12) and Storage target-wide (batch 12)** —
Storage's
previous index/-only inverse-selection flips to graded opt-outs. Cloud:
37
members -> 4 batches. Storage: 232 members -> 20 batches.
3. **Deduplicate format_v2 file-scope helpers** that clash under unity.
4. **Unity for the format_v2 half of Format (batch 8)** — the v1
`format/` tree is
slated for removal, so every v1 file stays individual rather than paying
the
v1/v2 twin-symbol dedup a joint unity would need.
5. **Resolve runtime/load/io/util file-scope clashes** — including two
more latent
bugs found by merged TUs: `column_predicate.h` had the same unqualified
`INTERNAL_ERROR` mis-binding as `merger.cpp`, and `vdatetime_value.h`
defined a
`static RE2` in the header (one RE2 construction per including TU at
startup;
now a C++17 inline variable, one program-wide). `runtime_profile.h` also
carried two never-referenced private fields; removed.
6. **Unity for Runtime, Load, IO and Util** (batch 12/12/16/24 by member
weight).
7. **Add missing include guards to two more headers**
(`cgroup_cpu_ctl.h`,
`complex_type_deserialize_util.h`) — found the same way as the two in
#66776.
8. **Unity for Core, Agent, Udf, Common, ann_index** — and the Service
negative
result recorded in-tree (see below).
9. **Unity for the thrift half of DorisGen (batch 8)** — protobuf `.cc`
files stay
individual structurally: protoc emits identical file-scope statics
(`schemas[]`, `file_default_instances[]`) in every one.
10. **Evict `file_cache_lru_tool.cpp` from `libIO.a`** — a standalone
tool with its
own `main()` must not be an archive member once unity can merge it into
an
object the `doris_be` link actually pulls in.
11. **Fail the configure when a unity skip entry goes stale** —
`set_source_files_properties()` silently ignores nonexistent paths, so a
rename would quietly dissolve a skip entry and the file would rejoin its
batch with no diagnostic. All thirteen skip lists (including the merged
Exec/Exprs and pilot ones) now go through
`doris_skip_unity_inclusion()`,
which FATAL_ERRORs on a nonexistent entry (generated `GENSRC_DIR` paths
exempt — they legitimately do not exist on a fresh configure).
### Benefit
Measured on the development branch this series is split from (arm64
macOS,
clang 20, `-j14`, PCH on, cold builds, same-machine paired runs):
- this wave (on top of the #66776 state): **7m57s -> 6m18s (-99s /
-20.7%)**,
sum-of-TU CPU (user) 98.7 -> 76.7 min, compile units 7810 -> 7280.
- static archives collapse as duplicate DWARF / weak-template copies
merge
(measured on this branch, unity OFF vs ON): libStorage.a 1592 -> 514 MB,
libCloud.a 441 -> 107 MB, libRuntime.a 356 -> 122 MB, libLoad.a 311 ->
100 MB, libFormat.a 1057 -> 760 MB, libCore.a 371 -> 182 MB. Less disk
thrash per build, faster archiving and linking.
- the four latent bugs above found and fixed (two of them wrong-code).
### Verification on this branch (arm64 macOS, clang 20, ENABLE_PCH=ON)
- unity=ON full build: zero failures across all 7927 TUs, `doris_be`
links
(319 MB). One upstream-drift clash surfaced and fixed first (the #66204
`path_allocated_bytes` twins — folded into the dedup commit).
- unity coverage after this PR: 110 batches / 1224 member files
tree-wide, of
which this PR adds 63 batches / 696 members (per-target counts in the
commit
messages); `compile_commands.json` agrees.
- `ENABLE_UNITY_BUILD=OFF` reconfigure: unity entries drop to zero and
the TU
count goes 7927 -> 9041 = exactly the 110 batches dissolving into their
1224
members; full rebuild green, `doris_be` relinks -> the switch's blast
radius
is precisely the unity config.
- BE UT (`BUILD_TYPE_UT=Debug`): doris_be_test compiles and links clean
— this
specifically exercises the four "a test #includes the src .cpp" opt-outs
below.
- fail-loud helper: deliberately renaming one skip entry makes the
configure
fail with `unity skip entry does not exist (renamed or moved?): <path>`
and
the offending CMakeLists line; reverting restores a clean configure.
### First CI round: two reds, both root-caused and fixed in-tree
- **ASAN Compile**: libstdc++-15 annotates loops in its own headers with
`#pragma GCC unroll`; under `-O1` + sanitizers + coverage the unroller
punts
and clang's `-Wpass-failed` (a missed-*hint* diagnostic with no
correctness
signal) became fatal under `-Werror` once a storage unity batch changed
the
emission context. Fixed by `-Wno-pass-failed` for clang tree-wide.
- **BE UT**: `FSFileCacheLeakCleanerTest.remove_orphan_and_tmp_files` —
a
pre-existing test-helper race: it waited for the meta-store *write
queue* to
drain, but the async worker dequeues before it issues the rocksdb Put,
so
the leak scan could still count zero metadata blocks and skip cleanup.
The
unity-batched IO objects shifted thread timing on the ASAN runner and
hit
the window (this test's only failure in its last 40 CI runs is this PR's
round — the race is upstream, the timing shift is ours). Fixed by
polling
the store itself instead of the queue size.
### Deliberate opt-outs (not omissions)
- **Per-file opt-outs** (~50 files across the targets, each annotated in
its
CMakeLists): file-scope macro leakers, the heaviest
template-instantiation TUs
(so no batch inherits their weight), `ENABLE_VARIANT_NESTED_GROUP` /
`ENABLE_TDE` module swaps (unaudited out-of-tree sources), and generated
protobuf `.cc`.
- **Four files a test compiles a second time by `#include`-ing the
.cpp**
(`storage/compaction/collection_statistics.cpp`,
`runtime/descriptors.cpp`,
`core/column/column_variant.cpp`,
`core/data_type/convert_field_to_type.cpp`):
the test object must shadow a never-pulled archive member, but a unity
batch
is pulled in for its siblings and the linker sees a duplicate
definition.
Same failure class the first CI round of #66776 hit; this time the whole
tree
was swept up front (`grep -rn '#include ".*\.cpp"' be/test`) and the
remaining
hits are either already opted out, structurally outside unity scope, or
test-side files.
- **Service stays at http-only unity**: widening to the whole target at
batch 8
was tried and measured **slower** (service segment slot time 104s ->
132s) —
splitting the single http jumbo repays the shared-closure parse per
batch and
the heterogeneous non-http TUs gain too little to cover it. The
CMakeLists
comment records this so nobody retries it blind.
- **contrib (openblas/clucene) deliberately untouched**: openblas is f2c
output —
2055 files with colliding `static c__1` file-scope constants,
structurally
un-unifiable.
### Cross-platform note (please watch the gcc pipeline)
All local verification is macOS/clang. gcc-only hazards were audited up
front:
the `-Wsubobject-linkage` pattern that bit #66776 (an
anonymous-namespace type as
a member of an external-linkage class turns into an error only once
unity makes
the file `#include`-d) was swept across all 627 files entering unity
scope —
every anonymous-namespace type usage found is function-local, so no
hazard — but
the sweep is heuristic, so the first Performance (gcc) CI round is worth
watching.
### Release note
None
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
- Full BE build with unity ON: all 7927 TUs green, `doris_be` links.
- `ENABLE_UNITY_BUILD=OFF` reconfigure + full rebuild green; TU
accounting closes exactly (7927 -> 9041); flip back ON restores.
- BE UT (`BUILD_TYPE_UT=Debug`): doris_be_test compiles and links with
zero duplicate symbols.
- Deliberately breaking a unity skip entry fails the configure with the
new fail-loud diagnostic; reverting restores a clean configure.
- Behavior changed:
- [x] Yes. Two latent wrong-code fixes surfaced by unity merges (details
in
the commit messages): `merger.cpp` and `column_predicate.h` threw with
`INTERNAL_ERROR` accidentally bound to `PPlanFragmentCancelReason` (=3);
they now carry `ErrorCode::INTERNAL_ERROR` as intended. Also
`vdatetime_value.h`'s header-defined `static RE2` became a C++17 inline
variable: one instance program-wide instead of one per including TU
(same matching behavior, less startup work and memory).
- Does this need documentation?
- [x] No.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Gdfkk7RqgD5e3Uv7bTM3NV
---------
Co-authored-by: Claude Fable 5 <[email protected]>
---
be/CMakeLists.txt | 27 +++++++++++++-
be/src/agent/CMakeLists.txt | 7 +++-
be/src/agent/cgroup_cpu_ctl.h | 2 ++
be/src/cloud/CMakeLists.txt | 12 +++++++
be/src/cloud/cloud_internal_service.cpp | 7 +---
be/src/cloud/cloud_warm_up_manager.cpp | 7 +---
be/src/common/CMakeLists.txt | 16 +++++++++
be/src/core/CMakeLists.txt | 38 ++++++++++++++++++++
.../complex_type_deserialize_util.h | 2 ++
be/src/core/value/vdatetime_value.h | 4 ++-
be/src/exec/CMakeLists.txt | 2 +-
be/src/exprs/CMakeLists.txt | 2 +-
be/src/format/CMakeLists.txt | 15 ++++++++
be/src/format_v2/column_mapper.cpp | 4 ---
be/src/format_v2/column_mapper_nested.cpp | 4 ---
be/src/format_v2/expr/cast.h | 5 +++
be/src/format_v2/parquet/parquet_scan.cpp | 9 +----
be/src/format_v2/parquet/parquet_statistics.cpp | 9 +----
be/src/format_v2/parquet/selection_vector.h | 8 +++++
be/src/format_v2/table/schema_history_util.cpp | 7 ----
be/src/format_v2/table/schema_history_util.h | 8 +++++
be/src/format_v2/table_reader.cpp | 8 +----
be/src/gen_cpp/CMakeLists.txt | 18 +++++++++-
be/src/io/CMakeLists.txt | 27 ++++++++++++++
be/src/io/cache/block_file_cache.cpp | 4 ---
be/src/io/cache/file_cache_common.h | 5 +++
be/src/io/cache/lru_queue_recorder.cpp | 8 +----
be/src/load/CMakeLists.txt | 10 ++++++
be/src/runtime/CMakeLists.txt | 14 ++++++++
be/src/runtime/runtime_profile.h | 3 --
be/src/service/CMakeLists.txt | 14 +++++---
be/src/storage/CMakeLists.txt | 41 +++++++++++++++-------
be/src/storage/index/ann/CMakeLists.txt | 5 +++
be/src/storage/merger.cpp | 20 +++++------
be/src/storage/predicate/column_predicate.h | 5 +--
be/src/storage/segment/common.h | 5 +++
be/src/storage/segment/external_col_meta_util.cpp | 6 ++--
be/src/storage/segment/segment.cpp | 2 +-
be/src/storage/segment/segment_writer.cpp | 4 +--
be/src/storage/segment/segment_writer.h | 3 --
.../segment/variant/v2/variant_path_builder.cpp | 4 ---
.../segment/variant/v2/variant_shredder.cpp | 5 +--
be/src/storage/segment/vertical_segment_writer.cpp | 4 +--
be/src/udf/CMakeLists.txt | 5 +++
be/src/util/CMakeLists.txt | 19 ++++++++++
be/src/util/bvar_windowed_adder.h | 6 ++++
be/src/util/json/path_in_data.h | 6 ++++
be/src/util/sm3.cpp | 6 ++--
be/src/util/timezone_utils.cpp | 6 ++--
.../fs_file_cache_storage_leak_cleaner_test.cpp | 6 ++--
50 files changed, 336 insertions(+), 128 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 0741d0a17dd..b2f1b8129ce 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -409,7 +409,14 @@ if (COMPILER_CLANG)
-Wthread-safety)
add_compile_options(-Wno-gnu-statement-expression
-Wno-implicit-float-conversion
- -Wno-sign-conversion)
+ -Wno-sign-conversion
+ # libstdc++-15 puts `#pragma GCC unroll` on loops in
its own
+ # headers (bits/stl_algobase.h); when a
sanitizer/coverage
+ # build compiles at low optimization the unroller
punts and
+ # clang's -Wpass-failed turns that missed *hint* into
an
+ # -Werror failure. The diagnostic carries no
correctness
+ # signal, so drop it.
+ -Wno-pass-failed)
if (USE_LIBCPP)
add_compile_options($<$<COMPILE_LANGUAGE:CXX>:-stdlib=libc++>)
add_definitions(-DUSE_LIBCPP)
@@ -981,6 +988,24 @@ function(pch_reuse target)
endif()
endfunction(pch_reuse target)
+# Every unity opt-out goes through this helper. set_source_files_properties()
+# silently ignores paths that do not exist, so a skip entry going stale after
+# a rename or move would quietly re-join its unity batch -- slower builds, or
+# new file-scope clashes, with no diagnostic anywhere. Fail the configure
+# instead. Entries under GENSRC_DIR are exempt: they are emitted by
+# add_custom_command at build time (wkt_lex.l.cpp, the generated
+# thrift/protobuf sources) and legitimately do not exist on a fresh configure.
+function(doris_skip_unity_inclusion)
+ foreach(entry IN LISTS ARGN)
+ string(FIND "${entry}" "${GENSRC_DIR}" gensrc_prefix_pos)
+ if(NOT gensrc_prefix_pos EQUAL 0 AND NOT EXISTS "${entry}")
+ message(FATAL_ERROR
+ "unity skip entry does not exist (renamed or moved?):
${entry}")
+ endif()
+ endforeach()
+ set_source_files_properties(${ARGN} PROPERTIES SKIP_UNITY_BUILD_INCLUSION
ON)
+endfunction()
+
add_subdirectory(${SRC_DIR}/agent)
add_subdirectory(${SRC_DIR}/common)
diff --git a/be/src/agent/CMakeLists.txt b/be/src/agent/CMakeLists.txt
index c680c41d6b2..423b9c5e1fb 100644
--- a/be/src/agent/CMakeLists.txt
+++ b/be/src/agent/CMakeLists.txt
@@ -29,4 +29,9 @@ endif()
add_library(Agent STATIC ${AGENT_SOURCES})
-pch_reuse(Agent)
\ No newline at end of file
+pch_reuse(Agent)
+
+# Unity build: nine homogeneous task-worker glue TUs sharing the agent header
+# closure; batch 0 merges them into a single unity TU.
+set_target_properties(Agent PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 0)
\ No newline at end of file
diff --git a/be/src/agent/cgroup_cpu_ctl.h b/be/src/agent/cgroup_cpu_ctl.h
index 262be57b499..3e1933fb4d6 100644
--- a/be/src/agent/cgroup_cpu_ctl.h
+++ b/be/src/agent/cgroup_cpu_ctl.h
@@ -15,6 +15,8 @@
// specific language governing permissions and limitations
// under the License.
+#pragma once
+
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
diff --git a/be/src/cloud/CMakeLists.txt b/be/src/cloud/CMakeLists.txt
index dbe8160c2f8..e81a097b1ff 100644
--- a/be/src/cloud/CMakeLists.txt
+++ b/be/src/cloud/CMakeLists.txt
@@ -22,3 +22,15 @@ file(GLOB_RECURSE SRC_FILES CONFIGURE_DEPENDS *.cpp)
add_library(Cloud STATIC ${SRC_FILES})
pch_reuse(Cloud)
+
+# Unity build: homogeneous cloud glue TUs sharing the CloudStorageEngine /
+# meta-service RPC header closure. Batch 12 bounds jumbo-TU size and memory.
+# The two http action TUs stay individual: sibling TUs' file-scope
+# `using namespace ErrorCode` makes the unscoped HttpStatus enum constants
+# (OK, NOT_FOUND, ...) shadow ErrorCode variables under -Wshadow -Werror.
+set(CLOUD_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/cloud_compaction_action.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/injection_point_action.cpp)
+doris_skip_unity_inclusion(${CLOUD_UNITY_SKIP})
+set_target_properties(Cloud PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 12)
diff --git a/be/src/cloud/cloud_internal_service.cpp
b/be/src/cloud/cloud_internal_service.cpp
index 0de9a78127a..d65e924fc4a 100644
--- a/be/src/cloud/cloud_internal_service.cpp
+++ b/be/src/cloud/cloud_internal_service.cpp
@@ -1002,12 +1002,7 @@ bvar::Adder<uint64_t>
g_file_cache_warm_up_rowset_wait_for_compaction_num(
bvar::Adder<uint64_t>
g_file_cache_warm_up_rowset_wait_for_compaction_timeout_num(
"file_cache_warm_up_rowset_wait_for_compaction_timeout_num");
-// Per-job windowed metrics for target BE
-// bvar::Window enforces MAX_SECONDS_LIMIT = 3600, so the longest window is 1h.
-static constexpr int WINDOW_5M = 300;
-static constexpr int WINDOW_30M = 1800;
-static constexpr int WINDOW_1H = 3600;
-
+// Per-job windowed metrics for target BE (window spans shared via
bvar_windowed_adder.h)
MBvarWindowedAdder
g_warmup_ed_finish_segment_num("warmup_ed_finish_segment_num", {"job_id"},
{WINDOW_5M, WINDOW_30M,
WINDOW_1H}, false);
MBvarWindowedAdder
g_warmup_ed_finish_segment_size("warmup_ed_finish_segment_size", {"job_id"},
diff --git a/be/src/cloud/cloud_warm_up_manager.cpp
b/be/src/cloud/cloud_warm_up_manager.cpp
index 4590d96fb4c..68db44f0b1c 100644
--- a/be/src/cloud/cloud_warm_up_manager.cpp
+++ b/be/src/cloud/cloud_warm_up_manager.cpp
@@ -118,12 +118,7 @@ bvar::Adder<int64_t>
g_file_cache_warm_up_job_num("file_cache_warm_up_job_num");
bvar::LatencyRecorder g_file_cache_warm_up_rowset_wait_for_compaction_latency(
"file_cache_warm_up_rowset_wait_for_compaction_latency");
-// Per-job windowed metrics for source BE
-// bvar::Window enforces MAX_SECONDS_LIMIT = 3600, so the longest window is 1h.
-static constexpr int WINDOW_5M = 300;
-static constexpr int WINDOW_30M = 1800;
-static constexpr int WINDOW_1H = 3600;
-
+// Per-job windowed metrics for source BE (window spans shared via
bvar_windowed_adder.h)
MBvarWindowedAdder
g_warmup_ed_requested_segment_num("warmup_ed_requested_segment_num", {"job_id"},
{WINDOW_5M, WINDOW_30M,
WINDOW_1H}, false);
MBvarWindowedAdder
g_warmup_ed_requested_segment_size("warmup_ed_requested_segment_size",
diff --git a/be/src/common/CMakeLists.txt b/be/src/common/CMakeLists.txt
index b1e51e8f303..949af9c9723 100644
--- a/be/src/common/CMakeLists.txt
+++ b/be/src/common/CMakeLists.txt
@@ -23,6 +23,22 @@ add_library(Common STATIC ${SRC_FILES})
pch_reuse(Common)
+# Unity build for the small common TUs. Seven files leak file-scope macros
+# (SET_FIELD/UPDATE_FIELD, MEMORY_SANITIZER, DW_* tables, the DEFINE_*_METRIC
+# families, USE_PHDR_CACHE/UNW_LOCAL_ONLY) and stay individual -- that set
+# also keeps the three same-name metric-hook statics (_s_hook_name) apart.
+set(COMMON_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/config.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/demangle.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/dwarf.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/metrics/doris_metrics.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/metrics/jvm_metrics.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/metrics/system_metrics.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/phdr_cache.cpp)
+doris_skip_unity_inclusion(${COMMON_UNITY_SKIP})
+set_target_properties(Common PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 0)
+
# Generate env_config.h according to env_config.h.in
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/env_config.h.in
${GENSRC_DIR}/common/env_config.h)
target_include_directories(Common PUBLIC ${GENSRC_DIR}/common/)
diff --git a/be/src/core/CMakeLists.txt b/be/src/core/CMakeLists.txt
index 348a375eef0..d5faec6c40c 100644
--- a/be/src/core/CMakeLists.txt
+++ b/be/src/core/CMakeLists.txt
@@ -32,3 +32,41 @@ pch_reuse(Core)
# instantiations below without definitions. Opt this single TU out of the PCH.
set_source_files_properties(${CMAKE_CURRENT_SOURCE_DIR}/wide_integer_from_double.cpp
PROPERTIES SKIP_PRECOMPILE_HEADERS ON)
+
+# Unity build: column/data_type/serde glue TUs share the core header closure.
+# Opt-outs:
+# - column/column_vector.cpp: the heaviest template-instantiation TU in the
+# target (~40s alone); it would dominate any batch it joins
+# - three macro-leak files (WRITE_INTEGRAL_COLUMN_TO_ORC,
+# DECLARE_DECIMAL_COMPARISON, DELARE_DATE_ADD_INTERVAL)
+# - wide_integer_from_double.cpp: macro-leak, and it is the PCH-skipped
+# explicit-instantiation TU above -- it must keep its own compile
+# - data_type_serde.cpp: holds member-level explicit instantiations of serde
+# members whose class-level explicit instantiations live in the per-type
+# serde TUs; merging both forms into one TU is a duplicate explicit
+# instantiation
+# - the value/variant/ family (9s of slot time in total): copied file-local
+# helpers with diverging signatures (require_bytes, write_unsigned, ...) --
+# not worth renaming for the gain
+# - two files that tests compile a second time by #including the .cpp
+# (column/column_variant.cpp, data_type/convert_field_to_type.cpp): the
+# test object must shadow a never-pulled archive member, but a unity batch
+# is pulled in for its siblings and the linker sees a duplicate definition
+set(CORE_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/column/column_vector.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/column/column_variant.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/data_type/convert_field_to_type.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/data_type_serde/data_type_number_serde.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/data_type_serde/data_type_serde.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/field.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/value/vdatetime_value.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/value/variant/variant_batch_builder.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/value/variant/variant_canonical.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/value/variant/variant_field.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/value/variant/variant_metadata.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/value/variant/variant_scalar.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/value/variant/variant_value.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/wide_integer_from_double.cpp)
+doris_skip_unity_inclusion(${CORE_UNITY_SKIP})
+set_target_properties(Core PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 12)
diff --git a/be/src/core/data_type_serde/complex_type_deserialize_util.h
b/be/src/core/data_type_serde/complex_type_deserialize_util.h
index ff76d64dcd6..b1320b4b073 100644
--- a/be/src/core/data_type_serde/complex_type_deserialize_util.h
+++ b/be/src/core/data_type_serde/complex_type_deserialize_util.h
@@ -15,6 +15,8 @@
// specific language governing permissions and limitations
// under the License.
+#pragma once
+
#include "core/data_type_serde/data_type_serde.h"
#include "core/string_ref.h"
diff --git a/be/src/core/value/vdatetime_value.h
b/be/src/core/value/vdatetime_value.h
index 7419b021755..da0bba19bcb 100644
--- a/be/src/core/value/vdatetime_value.h
+++ b/be/src/core/value/vdatetime_value.h
@@ -219,7 +219,9 @@ static constexpr uint32_t DATEV2_YEAR_WIDTH = 23;
static constexpr uint32_t DATETIMEV2_YEAR_WIDTH = 18;
static constexpr uint32_t DATETIMEV2_MONTH_WIDTH = 4;
-static RE2 time_zone_offset_format_reg(R"(^[+-]{1}\d{2}\:\d{2}$)");
+// One shared instance program-wide (was `static`, i.e. one copy constructed
+// per including TU). Visiting is thread-safe.
+inline RE2 time_zone_offset_format_reg(R"(^[+-]{1}\d{2}\:\d{2}$)");
uint8_t mysql_week_mode(uint32_t mode);
diff --git a/be/src/exec/CMakeLists.txt b/be/src/exec/CMakeLists.txt
index 7ec05378911..7ccbd60b5ea 100644
--- a/be/src/exec/CMakeLists.txt
+++ b/be/src/exec/CMakeLists.txt
@@ -52,7 +52,7 @@ set(EXEC_UNITY_SKIP
${CMAKE_CURRENT_SOURCE_DIR}/operator/partitioned_aggregation_sink_operator.cpp
${CMAKE_CURRENT_SOURCE_DIR}/operator/scan_operator.cpp
${CMAKE_CURRENT_SOURCE_DIR}/sink/writer/vfile_result_writer.cpp)
-set_source_files_properties(${EXEC_UNITY_SKIP} PROPERTIES
SKIP_UNITY_BUILD_INCLUSION ON)
+doris_skip_unity_inclusion(${EXEC_UNITY_SKIP})
set_target_properties(Exec PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
UNITY_BUILD_BATCH_SIZE 12)
diff --git a/be/src/exprs/CMakeLists.txt b/be/src/exprs/CMakeLists.txt
index b7bf6916a90..1940ba6d8e4 100644
--- a/be/src/exprs/CMakeLists.txt
+++ b/be/src/exprs/CMakeLists.txt
@@ -97,7 +97,7 @@ set(EXPRS_UNITY_SKIP
${CMAKE_CURRENT_SOURCE_DIR}/math_functions.cpp
${GENSRC_DIR}/geo/wkt_lex.l.cpp
${GENSRC_DIR}/geo/wkt_yacc.y.cpp)
-set_source_files_properties(${EXPRS_UNITY_SKIP} PROPERTIES
SKIP_UNITY_BUILD_INCLUSION ON)
+doris_skip_unity_inclusion(${EXPRS_UNITY_SKIP})
set_target_properties(Exprs PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
UNITY_BUILD_BATCH_SIZE 8)
diff --git a/be/src/format/CMakeLists.txt b/be/src/format/CMakeLists.txt
index f73ef2dbfaa..94b7c6bb6d1 100644
--- a/be/src/format/CMakeLists.txt
+++ b/be/src/format/CMakeLists.txt
@@ -29,3 +29,18 @@ list(APPEND SRC_FILES ${FORMAT_V2_SRC_FILES})
add_library(Format STATIC ${SRC_FILES})
pch_reuse(Format)
+
+# Unity build scoped to format_v2/: the v2 readers are the maintained line and
+# share one reader/expr header closure. The format/ (v1) tree is slated for
+# removal in a future version, so it is not worth the v1/v2 twin-symbol
+# deduplication a joint unity would require: every v1 file stays individual
+# until the tree is deleted. Within v2, adbc_reader.cpp stays individual
+# because its file-scope RETURN_IF_ADBC_ERROR macro must not leak into unity
+# siblings. Batch 8 bounds jumbo-TU size and memory.
+set(FORMAT_UNITY_SKIP ${SRC_FILES})
+list(FILTER FORMAT_UNITY_SKIP EXCLUDE REGEX ".*/format_v2/.*")
+list(APPEND FORMAT_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/../format_v2/table/adbc_reader.cpp)
+doris_skip_unity_inclusion(${FORMAT_UNITY_SKIP})
+set_target_properties(Format PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 8)
diff --git a/be/src/format_v2/column_mapper.cpp
b/be/src/format_v2/column_mapper.cpp
index 9f433cc853f..e98a0cde9a1 100644
--- a/be/src/format_v2/column_mapper.cpp
+++ b/be/src/format_v2/column_mapper.cpp
@@ -349,10 +349,6 @@ static VExprSPtr create_file_slot_ref(const VSlotRef&
slot_ref,
return ref;
}
-static bool is_cast_expr(const VExprSPtr& expr) {
- return dynamic_cast<const Cast*>(expr.get()) != nullptr;
-}
-
static bool is_binary_comparison_predicate(const VExprSPtr& expr) {
if (expr == nullptr || expr->get_num_children() != 2 ||
(expr->node_type() != TExprNodeType::BINARY_PRED &&
diff --git a/be/src/format_v2/column_mapper_nested.cpp
b/be/src/format_v2/column_mapper_nested.cpp
index fefcdf26f12..9d1980c3579 100644
--- a/be/src/format_v2/column_mapper_nested.cpp
+++ b/be/src/format_v2/column_mapper_nested.cpp
@@ -37,10 +37,6 @@ namespace doris::format {
namespace {
-static bool is_cast_expr(const VExprSPtr& expr) {
- return dynamic_cast<const Cast*>(expr.get()) != nullptr;
-}
-
static bool is_signed_integer_type(PrimitiveType type) {
switch (type) {
case TYPE_TINYINT:
diff --git a/be/src/format_v2/expr/cast.h b/be/src/format_v2/expr/cast.h
index 22604455e50..18b3bea676c 100644
--- a/be/src/format_v2/expr/cast.h
+++ b/be/src/format_v2/expr/cast.h
@@ -68,4 +68,9 @@ private:
std::string _expr_name;
FunctionBasePtr _function;
};
+
+inline bool is_cast_expr(const VExprSPtr& expr) {
+ return dynamic_cast<const Cast*>(expr.get()) != nullptr;
+}
+
} // namespace doris::format
diff --git a/be/src/format_v2/parquet/parquet_scan.cpp
b/be/src/format_v2/parquet/parquet_scan.cpp
index a9154021999..115ea2e373d 100644
--- a/be/src/format_v2/parquet/parquet_scan.cpp
+++ b/be/src/format_v2/parquet/parquet_scan.cpp
@@ -49,6 +49,7 @@
#include "format_v2/parquet/reader/native/column_chunk_reader.h"
#include "format_v2/parquet/reader/native_column_reader.h"
#include "format_v2/parquet/reader/row_position_column_reader.h"
+#include "format_v2/parquet/selection_vector.h" // count_range_rows
#include "runtime/runtime_state.h"
#include "util/defer_op.h"
#include "util/time.h"
@@ -823,14 +824,6 @@ Status execute_batch_filters(const
format::FileScanRequest& request, int64_t bat
}
namespace {
-int64_t count_range_rows(const std::vector<RowRange>& ranges) {
- int64_t rows = 0;
- for (const auto& range : ranges) {
- rows += range.length;
- }
- return rows;
-}
-
void append_intersection(const RowRange& left, const RowRange& right,
std::vector<RowRange>& result) {
const int64_t start = std::max(left.start, right.start);
diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp
b/be/src/format_v2/parquet/parquet_statistics.cpp
index e6f42c59ce6..b67536fd02c 100644
--- a/be/src/format_v2/parquet/parquet_statistics.cpp
+++ b/be/src/format_v2/parquet/parquet_statistics.cpp
@@ -42,6 +42,7 @@
#include "format_v2/parquet/parquet_file_context.h"
#include "format_v2/parquet/reader/native/block_split_bloom_filter.h"
#include "format_v2/parquet/reader/native_column_reader.h"
+#include "format_v2/parquet/selection_vector.h" // count_range_rows
#include "format_v2/timestamp_statistics.h"
#include "runtime/runtime_profile.h"
#include "storage/index/bloom_filter/bloom_filter.h"
@@ -989,14 +990,6 @@ std::vector<RowRange> intersect_ranges(const
std::vector<RowRange>& left,
return result;
}
-int64_t count_range_rows(const std::vector<RowRange>& ranges) {
- int64_t rows = 0;
- for (const auto& range : ranges) {
- rows += range.length;
- }
- return rows;
-}
-
void append_row_range(const RowRange& range, std::vector<RowRange>* ranges) {
if (range.length == 0) {
return;
diff --git a/be/src/format_v2/parquet/selection_vector.h
b/be/src/format_v2/parquet/selection_vector.h
index 99c854d8b4e..a9c8c040e93 100644
--- a/be/src/format_v2/parquet/selection_vector.h
+++ b/be/src/format_v2/parquet/selection_vector.h
@@ -31,6 +31,14 @@ struct RowRange {
int64_t length = 0;
};
+inline int64_t count_range_rows(const std::vector<RowRange>& ranges) {
+ int64_t rows = 0;
+ for (const auto& range : ranges) {
+ rows += range.length;
+ }
+ return rows;
+}
+
struct ParquetPageSkipPlan {
int leaf_column_id = -1;
// Page ordinal is the data-page ordinal in the column chunk. It
intentionally excludes
diff --git a/be/src/format_v2/table/schema_history_util.cpp
b/be/src/format_v2/table/schema_history_util.cpp
index 96a466fa772..a3dbad7c1e3 100644
--- a/be/src/format_v2/table/schema_history_util.cpp
+++ b/be/src/format_v2/table/schema_history_util.cpp
@@ -27,13 +27,6 @@
namespace doris::format {
namespace {
-const schema::external::TField* get_field_ptr(const
schema::external::TFieldPtr& field_ptr) {
- if (!field_ptr.__isset.field_ptr || field_ptr.field_ptr == nullptr) {
- return nullptr;
- }
- return field_ptr.field_ptr.get();
-}
-
const schema::external::TField* find_child_field_by_name(
const std::vector<schema::external::TFieldPtr>& fields, const
std::string& name) {
for (const auto& field_ptr : fields) {
diff --git a/be/src/format_v2/table/schema_history_util.h
b/be/src/format_v2/table/schema_history_util.h
index 3c4a80b5d4c..49704c5fb98 100644
--- a/be/src/format_v2/table/schema_history_util.h
+++ b/be/src/format_v2/table/schema_history_util.h
@@ -27,6 +27,14 @@
namespace doris::format {
+// Unwrap a thrift TFieldPtr, returning nullptr when the pointer is unset.
+inline const schema::external::TField* get_field_ptr(const
schema::external::TFieldPtr& field_ptr) {
+ if (!field_ptr.__isset.field_ptr || field_ptr.field_ptr == nullptr) {
+ return nullptr;
+ }
+ return field_ptr.field_ptr.get();
+}
+
const schema::external::TSchema* find_history_schema(const
TFileScanRangeParams* params,
int64_t schema_id);
diff --git a/be/src/format_v2/table_reader.cpp
b/be/src/format_v2/table_reader.cpp
index 40ffe6cb86e..b531ff9cf23 100644
--- a/be/src/format_v2/table_reader.cpp
+++ b/be/src/format_v2/table_reader.cpp
@@ -50,6 +50,7 @@
#include "format_v2/native/native_reader.h"
#include "format_v2/orc/orc_reader.h"
#include "format_v2/parquet/parquet_reader.h"
+#include "format_v2/table/schema_history_util.h" // get_field_ptr
#include "runtime/file_scan_profile.h"
#include "storage/segment/condition_cache.h"
#include "util/debug_points.h"
@@ -140,13 +141,6 @@ std::string partition_values_debug_string(const
std::map<std::string, Field>& pa
return out.str();
}
-const schema::external::TField* get_field_ptr(const
schema::external::TFieldPtr& field_ptr) {
- if (!field_ptr.__isset.field_ptr || field_ptr.field_ptr == nullptr) {
- return nullptr;
- }
- return field_ptr.field_ptr.get();
-}
-
const schema::external::TField* find_external_field_by_id(
const schema::external::TStructField* root, int32_t field_id) {
if (root == nullptr || !root->__isset.fields) {
diff --git a/be/src/gen_cpp/CMakeLists.txt b/be/src/gen_cpp/CMakeLists.txt
index c4463f75520..abbe3877079 100644
--- a/be/src/gen_cpp/CMakeLists.txt
+++ b/be/src/gen_cpp/CMakeLists.txt
@@ -27,10 +27,26 @@ list(FILTER SRC_FILES EXCLUDE REGEX
"_server\\.skeleton\\.cpp$")
add_compile_options(-Wno-return-type)
-# keep everything in one library, the object files reference
+# keep everything in one library, the object files reference
# each other
add_library(DorisGen STATIC ${SRC_FILES})
+# Unity build for the thrift-generated .cpp half (pre-scanned clean: no
+# file-scope macros or duplicate statics). Generated sources cannot be edited,
+# so anything that clashes can only be opted out, never fixed:
+# - every protobuf .cc stays individual: protoc emits the same file-scope
+# statics (schemas[], file_default_instances[]) in each one, a structural
+# clash for any merged pair
+# - the two heaviest thrift TUs stay individual so no batch inherits them
+set(GEN_UNITY_SKIP ${SRC_FILES})
+list(FILTER GEN_UNITY_SKIP INCLUDE REGEX "\\.cc$")
+list(APPEND GEN_UNITY_SKIP
+ ${GEN_CPP_DIR}/FrontendService_types.cpp
+ ${GEN_CPP_DIR}/PlanNodes_types.cpp)
+doris_skip_unity_inclusion(${GEN_UNITY_SKIP})
+set_target_properties(DorisGen PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 8)
+
if (COMPILER_CLANG)
target_compile_options(DorisGen PRIVATE
-Wno-implicit-int-conversion
diff --git a/be/src/io/CMakeLists.txt b/be/src/io/CMakeLists.txt
index 56c2eeb94a3..3026c82e11f 100644
--- a/be/src/io/CMakeLists.txt
+++ b/be/src/io/CMakeLists.txt
@@ -34,11 +34,38 @@ if(ENABLE_TDE)
endif()
list(REMOVE_ITEM IO_FILES
"${CMAKE_CURRENT_SOURCE_DIR}/fs/benchmark/fs_benchmark_tool.cpp")
+# Standalone tool with its own main(); must not be an archive member of
libIO.a,
+# or unity builds merge it into an object that the doris_be link actually
pulls in.
+list(REMOVE_ITEM IO_FILES
"${CMAKE_CURRENT_SOURCE_DIR}/cache/file_cache_lru_tool.cpp")
add_library(IO STATIC ${IO_FILES})
pch_reuse(IO)
+# Unity build for the io glue (small fs/cache TUs, parse-cost dominated).
+# Opt-outs:
+# - three filesystem TUs whose CHECK_*_CLIENT file-scope macros must not
+# leak into unity siblings
+# - http_file_system.cpp: sibling TUs' file-scope `using namespace ErrorCode`
+# makes the unscoped HttpStatus enum constants shadow ErrorCode variables
+# under -Wshadow -Werror (same pattern as the cloud http actions)
+# - benchmark sources and ENABLE_TDE module swaps stay individual
+set(IO_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/fs/broker_file_system.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/fs/hdfs_file_system.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/fs/s3_file_system.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/fs/http_file_system.cpp)
+file(GLOB_RECURSE IO_UNITY_BENCH_SRCS CONFIGURE_DEPENDS
+ ${CMAKE_CURRENT_SOURCE_DIR}/fs/benchmark/*.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/tools/*.cpp)
+list(APPEND IO_UNITY_SKIP ${IO_UNITY_BENCH_SRCS})
+if (ENABLE_TDE)
+ list(APPEND IO_UNITY_SKIP ${EXTRA_SOURCES})
+endif()
+doris_skip_unity_inclusion(${IO_UNITY_SKIP})
+set_target_properties(IO PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 16)
+
if (${BUILD_FS_BENCHMARK} STREQUAL "ON")
add_executable(fs_benchmark_tool
fs/benchmark/fs_benchmark_tool.cpp
diff --git a/be/src/io/cache/block_file_cache.cpp
b/be/src/io/cache/block_file_cache.cpp
index b44679a94fa..c4e7ef47853 100644
--- a/be/src/io/cache/block_file_cache.cpp
+++ b/be/src/io/cache/block_file_cache.cpp
@@ -66,10 +66,6 @@ namespace {
constexpr std::array<FileCacheType, 4> LRU_LOG_REPLAY_TYPES = {
FileCacheType::TTL, FileCacheType::INDEX, FileCacheType::NORMAL,
FileCacheType::DISPOSABLE};
-size_t file_cache_type_index(FileCacheType type) {
- return static_cast<size_t>(type);
-}
-
} // namespace
// Insert a block pointer into one shard while swallowing allocation failures.
diff --git a/be/src/io/cache/file_cache_common.h
b/be/src/io/cache/file_cache_common.h
index 8b52af8d161..3f8ca5e1ff0 100644
--- a/be/src/io/cache/file_cache_common.h
+++ b/be/src/io/cache/file_cache_common.h
@@ -43,6 +43,11 @@ enum FileCacheType {
DISPOSABLE = 0,
TTL = 3,
};
+
+inline size_t file_cache_type_index(FileCacheType type) {
+ return static_cast<size_t>(type);
+}
+
std::string cache_type_to_surfix(FileCacheType type);
FileCacheType surfix_to_cache_type(const std::string& str);
diff --git a/be/src/io/cache/lru_queue_recorder.cpp
b/be/src/io/cache/lru_queue_recorder.cpp
index 314a444e232..82edc5f6070 100644
--- a/be/src/io/cache/lru_queue_recorder.cpp
+++ b/be/src/io/cache/lru_queue_recorder.cpp
@@ -24,13 +24,7 @@
namespace doris::io {
-namespace {
-
-size_t file_cache_type_index(FileCacheType type) {
- return static_cast<size_t>(type);
-}
-
-} // namespace
+namespace {} // namespace
void LRUQueueRecorder::record_queue_event(FileCacheType type, CacheLRULogType
log_type,
const UInt128Wrapper hash, const
size_t offset,
diff --git a/be/src/load/CMakeLists.txt b/be/src/load/CMakeLists.txt
index 5c2e970f6d9..5faf360734f 100644
--- a/be/src/load/CMakeLists.txt
+++ b/be/src/load/CMakeLists.txt
@@ -25,3 +25,13 @@ file(GLOB_RECURSE SRC_FILES CONFIGURE_DEPENDS *.cpp)
add_library(Load STATIC ${SRC_FILES})
pch_reuse(Load)
+
+# Unity build: load channel/delta-writer/group-commit glue sharing one header
+# closure. Two files leak file-scope macros (UNKNOWN_ID_FOR_TEST,
+# HANDLE_ERROR/HANDLE_MULTI_TABLE_ERROR) and stay individual.
+set(LOAD_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/channel/load_stream.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/routine_load/routine_load_task_executor.cpp)
+doris_skip_unity_inclusion(${LOAD_UNITY_SKIP})
+set_target_properties(Load PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 12)
diff --git a/be/src/runtime/CMakeLists.txt b/be/src/runtime/CMakeLists.txt
index 7d54f9bd76b..0d2f3b567d2 100644
--- a/be/src/runtime/CMakeLists.txt
+++ b/be/src/runtime/CMakeLists.txt
@@ -34,3 +34,17 @@ add_library(Runtime STATIC
)
pch_reuse(Runtime)
+
+# Unity build: runtime glue TUs share the exec-env/runtime-state header
+# closure. memory/jemalloc_hook.cpp (a Linux-only member, absent on macOS)
+# stays individual: it defines malloc-replacement hooks and leaks
+# __THROW/ALIAS macros. descriptors.cpp stays individual: its test compiles
+# it a second time by #including the .cpp, and the test object must shadow a
+# never-pulled archive member (a unity batch would get pulled in for its
+# siblings and hand the linker a duplicate definition).
+set(RUNTIME_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/memory/jemalloc_hook.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/descriptors.cpp)
+doris_skip_unity_inclusion(${RUNTIME_UNITY_SKIP})
+set_target_properties(Runtime PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 12)
diff --git a/be/src/runtime/runtime_profile.h b/be/src/runtime/runtime_profile.h
index 54f2e80c89e..662a92db11d 100644
--- a/be/src/runtime/runtime_profile.h
+++ b/be/src/runtime/runtime_profile.h
@@ -746,9 +746,6 @@ private:
int64_t _metadata;
bool _is_set_metadata = false;
- bool _is_sink = false;
- bool _is_set_sink = false;
-
// The timestamp when the profile was modified, make sure the update is up
to date.
time_t _timestamp;
diff --git a/be/src/service/CMakeLists.txt b/be/src/service/CMakeLists.txt
index 28985a2e031..f07b896d4a4 100644
--- a/be/src/service/CMakeLists.txt
+++ b/be/src/service/CMakeLists.txt
@@ -37,16 +37,20 @@ add_library(Service STATIC ${SRC_FILES})
pch_reuse(Service)
# Unity build scoped to the http glue: ~60 small handlers re-parsing the same
-# header closure once per file. Non-http sources (service entry points,
-# arrow_flight) are heterogeneous heavy TUs that gain nothing from merging.
-# http_parser.cpp and be_thread_stack_action.cpp stay individual because their
-# file-scope macros (CR/LF, UNW_LOCAL_ONLY) must not leak into unity siblings.
+# header closure once per file, merged into one unity TU (batch 0). Widening
+# to the whole target at batch 8 was tried and measured slower: splitting the
+# single http jumbo into 8-file batches repays the shared-closure parse per
+# batch, and the heterogeneous non-http TUs gain too little to cover it
+# (service segment 104s -> 132s slot time). So non-http sources stay
+# individual. http_parser.cpp and be_thread_stack_action.cpp stay individual
+# because their file-scope macros (CR/LF, UNW_LOCAL_ONLY) must not leak into
+# unity siblings.
set(SERVICE_UNITY_SKIP ${SRC_FILES})
list(FILTER SERVICE_UNITY_SKIP EXCLUDE REGEX ".*/service/http/.*")
list(APPEND SERVICE_UNITY_SKIP
${CMAKE_CURRENT_SOURCE_DIR}/http/http_parser.cpp
${CMAKE_CURRENT_SOURCE_DIR}/http/action/be_thread_stack_action.cpp)
-set_source_files_properties(${SERVICE_UNITY_SKIP} PROPERTIES
SKIP_UNITY_BUILD_INCLUSION ON)
+doris_skip_unity_inclusion(${SERVICE_UNITY_SKIP})
set_target_properties(Service PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
UNITY_BUILD_BATCH_SIZE 0)
diff --git a/be/src/storage/CMakeLists.txt b/be/src/storage/CMakeLists.txt
index 888b16ebe32..21e82183f58 100644
--- a/be/src/storage/CMakeLists.txt
+++ b/be/src/storage/CMakeLists.txt
@@ -46,19 +46,36 @@ endif()
pch_reuse(Storage)
-# Unity build scoped to storage/index/: 115 homogeneous reader/writer/iterator
-# TUs sharing one (CLucene-heavy) header closure. The rest of Storage keeps
-# compiling individually. Batch 32 bounds jumbo-TU size and memory. Four index
-# files stay individual because their file-scope macros (CL_MAX_PATH and
-# friends, IS_CHINESE_CHAR, APPLY_FOR_PRIMITITYPE) must not leak into unity
-# siblings.
-set(STORAGE_UNITY_SKIP ${SRC_FILES})
-list(FILTER STORAGE_UNITY_SKIP EXCLUDE REGEX ".*/storage/index/.*")
-list(APPEND STORAGE_UNITY_SKIP
+# Unity build for the whole Storage target: segment/rowset/tablet/compaction/
+# index glue TUs re-parse the same storage header closure once per file, so
+# batching them cuts the repeated frontend work. Batch 12 bounds jumbo-TU size
+# and memory. Opted out:
+# - seven files whose file-scope macros (CL_MAX_PATH and friends,
+# IS_CHINESE_CHAR, APPLY_FOR_PRIMITITYPE, RETURN_IF_NOT_EOF_AND_OK,
+# FORMAT_*_ADD_JSON_NODE, RETURN_IF_ERROR_) must not leak into unity siblings
+# - the three heaviest template-instantiation TUs (predicate creators) which
+# would dominate any batch they join
+# - compaction/collection_statistics.cpp: its test compiles it a second time
+# by #including the .cpp; the test object must shadow a never-pulled archive
+# member, but a unity batch is pulled in for its siblings and the linker
+# sees a duplicate definition
+set(STORAGE_UNITY_SKIP
${CMAKE_CURRENT_SOURCE_DIR}/index/inverted/inverted_index_compound_reader.cpp
${CMAKE_CURRENT_SOURCE_DIR}/index/inverted/inverted_index_fs_directory.cpp
${CMAKE_CURRENT_SOURCE_DIR}/index/inverted/tokenizer/basic/basic_tokenizer.cpp
- ${CMAKE_CURRENT_SOURCE_DIR}/index/zone_map/zone_map_index.cpp)
-set_source_files_properties(${STORAGE_UNITY_SKIP} PROPERTIES
SKIP_UNITY_BUILD_INCLUSION ON)
+ ${CMAKE_CURRENT_SOURCE_DIR}/index/zone_map/zone_map_index.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/iterator/vcollect_iterator.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/tablet/tablet.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/task/engine_clone_task.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/predicate/predicate_creator_comparison.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/predicate/predicate_creator_in_list_in.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/predicate/predicate_creator_in_list_not_in.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/compaction/collection_statistics.cpp)
+if (ENABLE_VARIANT_NESTED_GROUP)
+ # Out-of-tree module sources swapped into this target: unity hygiene
+ # unaudited, keep them individual.
+ list(APPEND STORAGE_UNITY_SKIP ${VARIANT_NESTED_GROUP_SOURCES})
+endif()
+doris_skip_unity_inclusion(${STORAGE_UNITY_SKIP})
set_target_properties(Storage PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
- UNITY_BUILD_BATCH_SIZE 32)
+ UNITY_BUILD_BATCH_SIZE 12)
diff --git a/be/src/storage/index/ann/CMakeLists.txt
b/be/src/storage/index/ann/CMakeLists.txt
index a30bfce9773..d671cdbf024 100644
--- a/be/src/storage/index/ann/CMakeLists.txt
+++ b/be/src/storage/index/ann/CMakeLists.txt
@@ -30,3 +30,8 @@ target_link_libraries(ann_index PUBLIC faiss
OpenMP::OpenMP_CXX)
# Some header files from faiss are used by doris, they will break compile
check.
target_compile_options(ann_index PRIVATE -Wno-shadow-field)
+
+# Unity build: ten homogeneous TUs sharing the faiss header closure; batch 0
+# merges them into a single unity TU.
+set_target_properties(ann_index PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 0)
diff --git a/be/src/storage/merger.cpp b/be/src/storage/merger.cpp
index d986b367288..c8abc92e7f7 100644
--- a/be/src/storage/merger.cpp
+++ b/be/src/storage/merger.cpp
@@ -121,8 +121,8 @@ Status Merger::vmerge_rowsets(BaseTabletSPtr tablet,
ReaderType reader_type,
auto tablet_state = tablet->tablet_state();
if (tablet_state != TABLET_RUNNING && tablet_state != TABLET_NOTREADY)
{
tablet->clear_cache();
- return Status::Error<INTERNAL_ERROR>("tablet {} is not used any
more",
- tablet->tablet_id());
+ return Status::Error<ErrorCode::INTERNAL_ERROR>("tablet {} is not
used any more",
+
tablet->tablet_id());
}
// Read one block from block reader
@@ -144,8 +144,8 @@ Status Merger::vmerge_rowsets(BaseTabletSPtr tablet,
ReaderType reader_type,
block.clear_column_data();
}
if (ExecEnv::GetInstance()->storage_engine().stopped()) {
- return Status::Error<INTERNAL_ERROR>("tablet {} failed to do
compaction, engine stopped",
- tablet->tablet_id());
+ return Status::Error<ErrorCode::INTERNAL_ERROR>(
+ "tablet {} failed to do compaction, engine stopped",
tablet->tablet_id());
}
if (stats_output != nullptr) {
@@ -312,8 +312,8 @@ Status Merger::vertical_compact_one_group(
auto tablet_state = tablet->tablet_state();
if (tablet_state != TABLET_RUNNING && tablet_state != TABLET_NOTREADY)
{
tablet->clear_cache();
- return Status::Error<INTERNAL_ERROR>("tablet {} is not used any
more",
- tablet->tablet_id());
+ return Status::Error<ErrorCode::INTERNAL_ERROR>("tablet {} is not
used any more",
+
tablet->tablet_id());
}
// Read one block from block reader
RETURN_NOT_OK_STATUS_WITH_WARN(reader.next_block_with_aggregation(&block, &eof),
@@ -335,8 +335,8 @@ Status Merger::vertical_compact_one_group(
block.clear_column_data();
}
if (ExecEnv::GetInstance()->storage_engine().stopped()) {
- return Status::Error<INTERNAL_ERROR>("tablet {} failed to do
compaction, engine stopped",
- tablet->tablet_id());
+ return Status::Error<ErrorCode::INTERNAL_ERROR>(
+ "tablet {} failed to do compaction, engine stopped",
tablet->tablet_id());
}
if (stats_output != nullptr) {
@@ -391,8 +391,8 @@ Status Merger::vertical_compact_one_group(
block.clear_column_data();
}
if (ExecEnv::GetInstance()->storage_engine().stopped()) {
- return Status::Error<INTERNAL_ERROR>("tablet {} failed to do
compaction, engine stopped",
- tablet_id);
+ return Status::Error<ErrorCode::INTERNAL_ERROR>(
+ "tablet {} failed to do compaction, engine stopped",
tablet_id);
}
if (stats_output != nullptr) {
diff --git a/be/src/storage/predicate/column_predicate.h
b/be/src/storage/predicate/column_predicate.h
index 2bcc8359464..1d55531eb5e 100644
--- a/be/src/storage/predicate/column_predicate.h
+++ b/be/src/storage/predicate/column_predicate.h
@@ -349,7 +349,8 @@ public:
if (_predicate_input_rows_counter == nullptr ||
_predicate_filtered_rows_counter == nullptr ||
_predicate_always_true_rows_counter == nullptr) {
- throw Exception(INTERNAL_ERROR, "Predicate profile counters are
not initialized");
+ throw Exception(ErrorCode::INTERNAL_ERROR,
+ "Predicate profile counters are not initialized");
}
COUNTER_UPDATE(_predicate_input_rows_counter, input_rows);
COUNTER_UPDATE(_predicate_filtered_rows_counter, filter_rows);
@@ -397,7 +398,7 @@ public:
protected:
virtual bool _can_ignore() const { return _runtime_filter_id != -1; }
virtual uint16_t _evaluate_inner(const IColumn& column, uint16_t* sel,
uint16_t size) const {
- throw Exception(INTERNAL_ERROR, "Not Implemented _evaluate_inner");
+ throw Exception(ErrorCode::INTERNAL_ERROR, "Not Implemented
_evaluate_inner");
}
void reset_judge_selectivity() const {
_rf_selectivity.reset_judge_selectivity(); }
diff --git a/be/src/storage/segment/common.h b/be/src/storage/segment/common.h
index 166aa9fca58..a35fef0a97e 100644
--- a/be/src/storage/segment/common.h
+++ b/be/src/storage/segment/common.h
@@ -30,5 +30,10 @@ namespace segment_v2 {
using rowid_t = uint32_t;
using ordinal_t = uint64_t;
+// Magic footer bytes identifying a V2 segment file, written by the segment
+// writers and checked when parsing the footer.
+inline constexpr const char* k_segment_magic = "D0R1";
+inline constexpr uint32_t k_segment_magic_length = 4;
+
} // namespace segment_v2
} // namespace doris
diff --git a/be/src/storage/segment/external_col_meta_util.cpp
b/be/src/storage/segment/external_col_meta_util.cpp
index 87bc31f1079..9bc4bfeba13 100644
--- a/be/src/storage/segment/external_col_meta_util.cpp
+++ b/be/src/storage/segment/external_col_meta_util.cpp
@@ -29,8 +29,8 @@ namespace doris::segment_v2 {
namespace {
-io::IOContext create_index_io_context(const io::IOContext* source_io_ctx,
- OlapReaderStatistics* stats) {
+io::IOContext create_ext_meta_io_context(const io::IOContext* source_io_ctx,
+ OlapReaderStatistics* stats) {
io::IOContext io_ctx;
if (source_io_ctx != nullptr) {
io_ctx = *source_io_ctx;
@@ -145,7 +145,7 @@ Status ExternalColMetaUtil::read_col_meta(const
io::FileReaderSPtr& file_reader,
std::string buf;
buf.resize(static_cast<size_t>(size));
size_t meta_read = 0;
- io::IOContext io_ctx = create_index_io_context(source_io_ctx, stats);
+ io::IOContext io_ctx = create_ext_meta_io_context(source_io_ctx, stats);
RETURN_IF_ERROR(file_reader->read_at(pos, Slice(buf.data(), buf.size()),
&meta_read, &io_ctx));
if (meta_read != size) {
return Status::Corruption("short read ColumnMetaPB: expect={},
actual={}", size, meta_read);
diff --git a/be/src/storage/segment/segment.cpp
b/be/src/storage/segment/segment.cpp
index 9e58a76e0a1..be80f7ae530 100644
--- a/be/src/storage/segment/segment.cpp
+++ b/be/src/storage/segment/segment.cpp
@@ -73,11 +73,11 @@
#include "storage/segment/column_meta_accessor.h"
#include "storage/segment/column_reader.h"
#include "storage/segment/column_reader_cache.h"
+#include "storage/segment/common.h" // k_segment_magic
#include "storage/segment/empty_segment_iterator.h"
#include "storage/segment/page_io.h"
#include "storage/segment/page_pointer.h"
#include "storage/segment/segment_iterator.h"
-#include "storage/segment/segment_writer.h" // k_segment_magic_length
#include "storage/segment/stream_reader.h"
#include "storage/segment/variant/variant_column_reader.h"
#include "storage/tablet/tablet_schema.h"
diff --git a/be/src/storage/segment/segment_writer.cpp
b/be/src/storage/segment/segment_writer.cpp
index ed2b403c75f..2bdefe39ef7 100644
--- a/be/src/storage/segment/segment_writer.cpp
+++ b/be/src/storage/segment/segment_writer.cpp
@@ -62,6 +62,7 @@
#include "storage/rowset/rowset_writer_context.h" // RowsetWriterContext
#include "storage/rowset/segment_creator.h"
#include "storage/segment/column_writer.h" // ColumnWriter
+#include "storage/segment/common.h" // k_segment_magic
#include "storage/segment/encoding_info.h"
#include "storage/segment/external_col_meta_util.h"
#include "storage/segment/page_io.h"
@@ -80,9 +81,6 @@ namespace segment_v2 {
using namespace ErrorCode;
-const char* k_segment_magic = "D0R1";
-const uint32_t k_segment_magic_length = 4;
-
inline std::string segment_mem_tracker_name(uint32_t segment_id) {
return "SegmentWriter:Segment-" + std::to_string(segment_id);
}
diff --git a/be/src/storage/segment/segment_writer.h
b/be/src/storage/segment/segment_writer.h
index be22adcefaa..ba5d2f9af44 100644
--- a/be/src/storage/segment/segment_writer.h
+++ b/be/src/storage/segment/segment_writer.h
@@ -60,9 +60,6 @@ class FileWriter;
namespace segment_v2 {
-extern const char* k_segment_magic;
-extern const uint32_t k_segment_magic_length;
-
class VariantStatsCaculator;
struct SegmentWriterOptions {
diff --git a/be/src/storage/segment/variant/v2/variant_path_builder.cpp
b/be/src/storage/segment/variant/v2/variant_path_builder.cpp
index c7542669fe4..d908963ef74 100644
--- a/be/src/storage/segment/variant/v2/variant_path_builder.cpp
+++ b/be/src/storage/segment/variant/v2/variant_path_builder.cpp
@@ -849,10 +849,6 @@ size_t dotted_path_depth(const PathInData& path) {
return path.get_parts().size();
}
-size_t path_allocated_bytes(const PathInData& path) {
- return path.get_path().capacity() + path.get_parts().capacity() *
sizeof(PathInData::Part);
-}
-
size_t recursive_null_count(const IColumn& column) {
if (const auto* nullable = check_and_get_column<ColumnNullable>(column)) {
size_t count = 0;
diff --git a/be/src/storage/segment/variant/v2/variant_shredder.cpp
b/be/src/storage/segment/variant/v2/variant_shredder.cpp
index d61b51aed96..99c312a8326 100644
--- a/be/src/storage/segment/variant/v2/variant_shredder.cpp
+++ b/be/src/storage/segment/variant/v2/variant_shredder.cpp
@@ -35,6 +35,7 @@
#include "exec/common/variant_util.h"
#include "exprs/function/parse/variant_jsonb_parse.h"
#include "storage/tablet/tablet_schema.h"
+#include "util/json/path_in_data.h"
#include "util/jsonb_writer.h"
namespace doris::segment_v2 {
@@ -52,10 +53,6 @@ PathInData normalize_doc_publication_path(const PathInData&
path) {
return PathInData(path.get_path(), path.get_is_typed());
}
-size_t path_allocated_bytes(const PathInData& path) {
- return path.get_path().capacity() + path.get_parts().capacity() *
sizeof(PathInData::Part);
-}
-
} // namespace
struct VariantShredder::Impl {
diff --git a/be/src/storage/segment/vertical_segment_writer.cpp
b/be/src/storage/segment/vertical_segment_writer.cpp
index abbdb857bc4..319597a0418 100644
--- a/be/src/storage/segment/vertical_segment_writer.cpp
+++ b/be/src/storage/segment/vertical_segment_writer.cpp
@@ -68,6 +68,7 @@
#include "storage/rowset/rowset_writer_context.h" // RowsetWriterContext
#include "storage/rowset/segment_creator.h"
#include "storage/segment/column_writer.h" // ColumnWriter
+#include "storage/segment/common.h" // k_segment_magic
#include "storage/segment/encoding_info.h"
#include "storage/segment/external_col_meta_util.h"
#include "storage/segment/historical_row_retriever.h"
@@ -88,9 +89,6 @@ namespace doris::segment_v2 {
using namespace ErrorCode;
-static constexpr const char* k_segment_magic = "D0R1";
-static constexpr uint32_t k_segment_magic_length = 4;
-
inline std::string vertical_segment_writer_mem_tracker_name(uint32_t
segment_id) {
return "VerticalSegmentWriter:Segment-" + std::to_string(segment_id);
}
diff --git a/be/src/udf/CMakeLists.txt b/be/src/udf/CMakeLists.txt
index 0d57e85f5b1..45e28f93cb4 100755
--- a/be/src/udf/CMakeLists.txt
+++ b/be/src/udf/CMakeLists.txt
@@ -26,3 +26,8 @@ file(GLOB PYTHON_UDF_SOURCES "python/*.cpp")
# can have dependencies on our other libs. The second version is shipped as
part
# of the UDF sdk, which can't use other libs.
add_library(Udf STATIC ${PYTHON_UDF_SOURCES})
+
+# Unity build: eight homogeneous python-UDF glue TUs sharing the Python/arrow
+# header closure; batch 0 merges them into a single unity TU.
+set_target_properties(Udf PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 0)
diff --git a/be/src/util/CMakeLists.txt b/be/src/util/CMakeLists.txt
index f23c3ad865c..8941a7ed57f 100644
--- a/be/src/util/CMakeLists.txt
+++ b/be/src/util/CMakeLists.txt
@@ -41,3 +41,22 @@ add_library(Util STATIC
)
pch_reuse(Util)
+
+# Unity build for the small util TUs (parse-overhead dominated). Ten files
+# leak file-scope macros and stay individual; hash/city.cc keeps its generic
+# k0..k4 static hash constants away from sibling scopes. perf_counters.cpp
+# is a Linux-only member (macOS builds its _mac variant, which is clean).
+set(UTIL_UNITY_SKIP
+ ${CMAKE_CURRENT_SOURCE_DIR}/bfd_parser.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/hash/city.cc
+ ${CMAKE_CURRENT_SOURCE_DIR}/hash/murmur_hash3.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/network_util.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/os_info.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/os_util.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/perf_counters.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/simd/parquet_kernels.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/thrift_util.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/zlib.cpp)
+doris_skip_unity_inclusion(${UTIL_UNITY_SKIP})
+set_target_properties(Util PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
+ UNITY_BUILD_BATCH_SIZE 24)
diff --git a/be/src/util/bvar_windowed_adder.h
b/be/src/util/bvar_windowed_adder.h
index c4e9245b7e3..f249dc3a198 100644
--- a/be/src/util/bvar_windowed_adder.h
+++ b/be/src/util/bvar_windowed_adder.h
@@ -33,6 +33,12 @@
namespace doris {
+// Shared window spans (seconds) for multi-window metrics.
+// bvar::Window enforces MAX_SECONDS_LIMIT = 3600, so the longest window is 1h.
+inline constexpr int WINDOW_5M = 300;
+inline constexpr int WINDOW_30M = 1800;
+inline constexpr int WINDOW_1H = 3600;
+
/**
* Multi-dimension windowed adder.
*
diff --git a/be/src/util/json/path_in_data.h b/be/src/util/json/path_in_data.h
index 593904dd931..2e549cb714a 100644
--- a/be/src/util/json/path_in_data.h
+++ b/be/src/util/json/path_in_data.h
@@ -135,6 +135,12 @@ private:
bool is_typed = false;
};
+// Heap bytes retained beyond sizeof(PathInData): the dotted-path string buffer
+// plus the parts vector.
+inline size_t path_allocated_bytes(const PathInData& path) {
+ return path.get_path().capacity() + path.get_parts().capacity() *
sizeof(PathInData::Part);
+}
+
class PathInDataBuilder {
public:
const PathInData::Parts& get_parts() const { return parts; }
diff --git a/be/src/util/sm3.cpp b/be/src/util/sm3.cpp
index 2d4349abc84..448f2365e5f 100644
--- a/be/src/util/sm3.cpp
+++ b/be/src/util/sm3.cpp
@@ -46,11 +46,11 @@ void SM3Digest::digest() {
char hex_buf[2 * SM3_DIGEST_LENGTH];
- static char dig_vec_lower[] = "0123456789abcdef";
+ static constexpr char sm3_dig_vec_lower[] = "0123456789abcdef";
char* to = hex_buf;
for (int i = 0; i < SM3_DIGEST_LENGTH; ++i) {
- *to++ = dig_vec_lower[buf[i] >> 4];
- *to++ = dig_vec_lower[buf[i] & 0x0F];
+ *to++ = sm3_dig_vec_lower[buf[i] >> 4];
+ *to++ = sm3_dig_vec_lower[buf[i] & 0x0F];
}
_hex.assign(hex_buf, 2 * SM3_DIGEST_LENGTH);
}
diff --git a/be/src/util/timezone_utils.cpp b/be/src/util/timezone_utils.cpp
index b575ed7a29f..358d8b962cc 100644
--- a/be/src/util/timezone_utils.cpp
+++ b/be/src/util/timezone_utils.cpp
@@ -51,7 +51,7 @@ namespace doris {
using ZoneList = std::unordered_map<std::string, cctz::time_zone>;
-RE2 time_zone_offset_format_reg(R"(^[+-]{1}\d{2}\:\d{2}$)"); // visiting is
thread-safe
+static RE2 tz_offset_format_reg(R"(^[+-]{1}\d{2}\:\d{2}$)"); // visiting is
thread-safe
// for ut, make it never nullptr.
std::unique_ptr<ZoneList> lower_zone_cache_ = std::make_unique<ZoneList>();
@@ -277,8 +277,8 @@ bool TimezoneUtils::parse_tz_offset_string(const
std::string& timezone, cctz::ti
}
re2::StringPiece value;
- if (time_zone_offset_format_reg.Match(normalized, 0, normalized.size(),
RE2::UNANCHORED, &value,
- 1)) [[likely]] {
+ if (tz_offset_format_reg.Match(normalized, 0, normalized.size(),
RE2::UNANCHORED, &value, 1))
+ [[likely]] {
const bool positive = value[0] != '-';
const int hour = std::stoi(value.substr(1, 2).as_string());
const int minute = std::stoi(value.substr(4, 2).as_string());
diff --git a/be/test/io/cache/fs_file_cache_storage_leak_cleaner_test.cpp
b/be/test/io/cache/fs_file_cache_storage_leak_cleaner_test.cpp
index a4b5035ad45..0ab9953cb54 100644
--- a/be/test/io/cache/fs_file_cache_storage_leak_cleaner_test.cpp
+++ b/be/test/io/cache/fs_file_cache_storage_leak_cleaner_test.cpp
@@ -131,8 +131,10 @@ protected:
BlockMetaKey mkey(0, hash, offset);
BlockMeta meta(FileCacheType::NORMAL, 16, 0);
storage._meta_store->put(mkey, meta);
- // Wait for async write to complete for test stability
- for (int i = 0; i < 100 &&
storage._meta_store->get_write_queue_size() > 0; ++i) {
+ // The async worker dequeues an operation before it lands in
rocksdb,
+ // so an empty write queue does not mean the entry is readable yet.
+ // Poll the store itself: leak-scan decisions count rocksdb
contents.
+ for (int i = 0; i < 100 &&
!storage._meta_store->get(mkey).has_value(); ++i) {
std::this_thread::sleep_for(std::chrono::milliseconds(50));
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
