This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2a3a3908809 [opt](build) Decouple common/status.h from thrift/protobuf
generated headers (#67036)
2a3a3908809 is described below
commit 2a3a3908809c9072bf60c4838ec7d83f9b4ab776
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Sep 2 15:07:46 2026 +0800
[opt](build) Decouple common/status.h from thrift/protobuf generated
headers (#67036)
### What problem does this PR solve?
Issue Number: none
Related PR: #66901 (header-hygiene gate this PR extends), #66715 (BE
build-speed umbrella, closed)
Problem Summary:
`common/status.h` is included by ~2450 of ~2500 BE TUs and carried
`gen_cpp/Status_types.h` (the whole thrift runtime) plus
`gen_cpp/types.pb.h`
(the protobuf runtime) into every one of them. For any TU whose only
path to
the generated headers was status.h, that is ~44k preprocessed lines of
pure
tax per TU — paid in every build without a usable PCH: the Linux BE UT
line
(be/test builds with no PCH), `ENABLE_PCH=OFF` (gcc) builds,
clangd/clang-tidy,
and every incremental build after touching
`Status.thrift`/`types.proto`.
This PR decouples status.h/exception.h (and `util/hash_util.hpp`) from
the
generated headers, with **zero runtime semantic change**:
1. **status.h**: `APPLY_FOR_THRIFT_ERROR_CODES` now carries the literal
enum
value per entry; `status.cpp` locks every value to the thrift enum with
a
`static_assert`, so `Status.thrift` stays the single source of truth —
any
drift is a compile error. `Status::create<stacktrace>(TStatus/PStatus)`
move out of line with 4 explicit instantiations (cold path: RPC error
conversion), `code_as_string()` moves out of line (cold path: error
printing), and both generated includes are dropped (`TStatus` joins
`PStatus` as a forward declaration).
2. **exception.h**: drops its unused `Status_types.h` include.
3. **hash_util.hpp**: the `std::hash<TUniqueId/TNetworkAddress/pair>`
specializations keep only their declarations here (this header stays the
earliest carrier, so they remain visible before any implicit
instantiation); the bodies move to `uid_util.cpp`/`network_util.cpp`
where
the types are complete. The `Types_types.h` include is then dropped.
4. Mechanical fallout, all preparation-wave commits: `endian.h` gets its
own
`gbswap_16/32` builtins instead of leaning on the `bswap_*` macros that
protobuf's `stubs/port.h` happened to leak into most TUs;
`IDataType::to_protobuf(PTypeDesc*)` and the nine derived overrides move
out of line (data_type.h forward-declares the pb types); every TU that
named `TStatusCode`/`TStatus`/`PStatus`/`PValues` or `std::set`/
`std::unordered_set` via ride-along gets its direct include.
**Numbers** (clang -E, macOS arm; text include-graph reach over
be/src+be/test):
| metric | before | after |
|---|---|---|
| `common/status.h` closure | 147,838 lines | **103,883 (-29.7%)** |
| `util/hash_util.hpp` closure | 178,644 lines | **131,801 (-26.2%)** |
| `util/bitmap.cpp` natural closure (representative winner) | 172,721 |
**128,752 (-25%)** |
| `gen_cpp/Status_types.h` reach | 2453 TUs | **37 TUs** |
| `gen_cpp/types.pb.h` reach | 2458 TUs | 1791 TUs |
Cold build wall-clock with clang + `ENABLE_PCH=ON` is **neutral**
(687.3s ->
693.6s, +0.9%, paired back-to-back `--compile-bench -j 6` runs, within
run-to-run noise) — expected, since pch.h itself still includes the
generated
headers. The wins are the no-PCH worlds and the incremental radius:
editing
`Status.thrift` no longer invalidates essentially every TU there.
**Guards** (extends #66901): `check-header-deps.py` now forbids
status.h/exception.h/hash_util.hpp from reaching `gen_cpp/*` (empty
whitelists), the exec_env gen_cpp whitelist empties out, thread_context
loses
its `Status_types.h` entry, and `REVERSE_REACH_BASELINES` pins the three
generated headers at their new reach so the tax cannot silently regrow.
`syntax_sweep.py` gains `--include-tests` so the UT line's natural
closures
are sweepable.
---
be/src/agent/be_exec_version_manager.h | 4 +
be/src/agent/cgroup_cpu_ctl.cpp | 1 +
be/src/agent/cgroup_cpu_ctl.h | 1 +
be/src/cloud/cloud_backend_service.cpp | 1 +
be/src/cloud/cloud_internal_service.cpp | 1 +
be/src/cloud/cloud_meta_mgr.cpp | 1 +
be/src/common/exception.h | 1 -
be/src/common/status.cpp | 33 ++++++
be/src/common/status.h | 129 ++++++++++-----------
be/src/core/data_type/data_type.cpp | 8 ++
be/src/core/data_type/data_type.h | 13 +--
be/src/core/data_type/data_type_array.cpp | 7 ++
be/src/core/data_type/data_type_array.h | 6 +-
.../data_type/data_type_date_or_datetime_v2.cpp | 6 +
.../core/data_type/data_type_date_or_datetime_v2.h | 4 +-
be/src/core/data_type/data_type_decimal.cpp | 9 ++
be/src/core/data_type/data_type_decimal.h | 5 +-
be/src/core/data_type/data_type_map.cpp | 10 ++
be/src/core/data_type/data_type_map.h | 8 +-
be/src/core/data_type/data_type_string.cpp | 8 ++
be/src/core/data_type/data_type_string.h | 6 +-
be/src/core/data_type/data_type_struct.cpp | 14 +++
be/src/core/data_type/data_type_struct.h | 12 +-
be/src/core/data_type/data_type_timestamptz.cpp | 6 +
be/src/core/data_type/data_type_timestamptz.h | 4 +-
be/src/core/data_type/data_type_varbinary.cpp | 6 +
be/src/core/data_type/data_type_varbinary.h | 4 +-
be/src/core/data_type/data_type_variant.cpp | 8 ++
be/src/core/data_type/data_type_variant.h | 6 +-
.../core/data_type_serde/data_type_array_serde.cpp | 1 +
.../core/data_type_serde/data_type_map_serde.cpp | 2 +
.../data_type_serde/data_type_struct_serde.cpp | 2 +
be/src/exec/common/endian.h | 29 +++--
be/src/exec/scan/file_scanner.cpp | 1 +
be/src/exec/scan/split_source_connector.cpp | 2 +
be/src/exec/sink/vrow_distribution.cpp | 1 +
be/src/runtime/fragment_mgr.cpp | 1 +
be/src/service/backend_service.h | 1 +
be/src/service/http/http_handler_with_auth.cpp | 1 +
be/src/storage/index/inverted/setting.h | 1 +
be/src/storage/olap_server.cpp | 1 +
be/src/util/hash_util.hpp | 35 +++---
be/src/util/network_util.cpp | 8 ++
be/src/util/timezone_utils.cpp | 1 +
be/src/util/uid_util.cpp | 16 +++
be/test/agent/heartbeat_server_test.cpp | 1 +
be/test/common/status_test.cpp | 1 +
.../data_type_serde_fixed_length_object_test.cpp | 1 +
.../runtime_filter/sync_size_callback_test.cpp | 1 +
.../schema_tso_status_scanner_test.cpp | 1 +
be/test/exec/sink/vrow_distribution_test.cpp | 1 +
.../cache/cached_remote_file_reader_peer_test.cpp | 1 +
.../io/fs/rate_limited_obj_storage_client_test.cpp | 1 +
be/test/io/fs/s3_file_writer_test.cpp | 1 +
be/test/runtime/load_stream_test.cpp | 2 +
.../bloom_filter_index_reader_writer_test.cpp | 1 +
be/test/storage/segment/bloom_filter_test.cpp | 1 +
.../testutil/mock/obj_storage_client_test_stub.h | 2 +
build-support/check-header-deps.py | 55 +++++++--
build-support/compile-bench/syntax_sweep.py | 18 ++-
60 files changed, 344 insertions(+), 169 deletions(-)
diff --git a/be/src/agent/be_exec_version_manager.h
b/be/src/agent/be_exec_version_manager.h
index a5f8ac9ced3..2d7735518c0 100644
--- a/be/src/agent/be_exec_version_manager.h
+++ b/be/src/agent/be_exec_version_manager.h
@@ -20,6 +20,10 @@
#include <fmt/format.h>
#include <glog/logging.h>
+#include <map>
+#include <set>
+#include <string>
+
#include "common/exception.h"
#include "common/status.h"
diff --git a/be/src/agent/cgroup_cpu_ctl.cpp b/be/src/agent/cgroup_cpu_ctl.cpp
index 936eabda4d1..a39d6f5d5a7 100644
--- a/be/src/agent/cgroup_cpu_ctl.cpp
+++ b/be/src/agent/cgroup_cpu_ctl.cpp
@@ -22,6 +22,7 @@
#include <unistd.h>
#include <filesystem>
+#include <set>
#include "util/cgroup_util.h"
#include "util/defer_op.h"
diff --git a/be/src/agent/cgroup_cpu_ctl.h b/be/src/agent/cgroup_cpu_ctl.h
index 3e1933fb4d6..38f3e53f15d 100644
--- a/be/src/agent/cgroup_cpu_ctl.h
+++ b/be/src/agent/cgroup_cpu_ctl.h
@@ -22,6 +22,7 @@
#include <sys/types.h>
#include <unistd.h>
+#include <set>
#include <shared_mutex>
#include "common/config.h"
diff --git a/be/src/cloud/cloud_backend_service.cpp
b/be/src/cloud/cloud_backend_service.cpp
index 5f09f194e37..e605305b2c8 100644
--- a/be/src/cloud/cloud_backend_service.cpp
+++ b/be/src/cloud/cloud_backend_service.cpp
@@ -18,6 +18,7 @@
#include "cloud/cloud_backend_service.h"
#include <brpc/controller.h>
+#include <gen_cpp/Status_types.h>
#include "cloud/cloud_storage_engine.h"
#include "cloud/cloud_tablet.h"
diff --git a/be/src/cloud/cloud_internal_service.cpp
b/be/src/cloud/cloud_internal_service.cpp
index a9cb5a0b58b..4c0fd49b8d1 100644
--- a/be/src/cloud/cloud_internal_service.cpp
+++ b/be/src/cloud/cloud_internal_service.cpp
@@ -21,6 +21,7 @@
#include <bthread/countdown_event.h>
#include <butil/iobuf.h>
#include <fmt/format.h>
+#include <gen_cpp/Status_types.h>
#include <algorithm>
#include <chrono>
diff --git a/be/src/cloud/cloud_meta_mgr.cpp b/be/src/cloud/cloud_meta_mgr.cpp
index 6533734d00c..4bf1ea756e0 100644
--- a/be/src/cloud/cloud_meta_mgr.cpp
+++ b/be/src/cloud/cloud_meta_mgr.cpp
@@ -25,6 +25,7 @@
#include <gen_cpp/FrontendService.h>
#include <gen_cpp/HeartbeatService_types.h>
#include <gen_cpp/PlanNodes_types.h>
+#include <gen_cpp/Status_types.h>
#include <gen_cpp/Types_types.h>
#include <gen_cpp/cloud.pb.h>
#include <gen_cpp/olap_file.pb.h>
diff --git a/be/src/common/exception.h b/be/src/common/exception.h
index e03286c20e3..a823bcfdb82 100644
--- a/be/src/common/exception.h
+++ b/be/src/common/exception.h
@@ -18,7 +18,6 @@
#pragma once
#include <fmt/format.h>
-#include <gen_cpp/Status_types.h>
#include <exception>
#include <memory>
diff --git a/be/src/common/status.cpp b/be/src/common/status.cpp
index cc6c10c2941..8719ea7aa25 100644
--- a/be/src/common/status.cpp
+++ b/be/src/common/status.cpp
@@ -19,10 +19,43 @@
namespace doris {
namespace ErrorCode {
+// Pairing lock between the literal values in APPLY_FOR_THRIFT_ERROR_CODES and
+// the thrift enum generated from Status.thrift: if either side drifts, this
+// translation unit fails to compile.
+#define M(NAME, VALUE, ENABLESTACKTRACE) \
+ static_assert(static_cast<int>(TStatusCode::NAME) == (VALUE), \
+ "ErrorCode::" #NAME " drifted from Status.thrift");
+APPLY_FOR_THRIFT_ERROR_CODES(M)
+#undef M
+
ErrorCodeState error_states[MAX_ERROR_CODE_DEFINE_NUM];
ErrorCodeInitializer error_code_init(10);
} // namespace ErrorCode
+template <bool stacktrace>
+Status Status::create(const TStatus& status) {
+ return Error<stacktrace>(status.status_code,
+ "TStatus: " + (status.error_msgs.empty() ? "" :
status.error_msgs[0]));
+}
+
+template Status Status::create<true>(const TStatus& status);
+template Status Status::create<false>(const TStatus& status);
+
+template <bool stacktrace>
+Status Status::create(const PStatus& pstatus) {
+ return Error<stacktrace>(
+ pstatus.status_code(),
+ "PStatus: " + (pstatus.error_msgs_size() == 0 ? "" :
pstatus.error_msgs(0)));
+}
+
+template Status Status::create<true>(const PStatus& pstatus);
+template Status Status::create<false>(const PStatus& pstatus);
+
+std::string Status::code_as_string() const {
+ return (int)_code >= 0 ?
doris::to_string(static_cast<TStatusCode::type>(_code))
+ : fmt::format("E{}", (int16_t)_code);
+}
+
void Status::to_thrift(TStatus* s) const {
s->error_msgs.clear();
if (ok()) {
diff --git a/be/src/common/status.h b/be/src/common/status.h
index 1e4a0de46f8..11bd06644f0 100644
--- a/be/src/common/status.h
+++ b/be/src/common/status.h
@@ -5,8 +5,6 @@
#pragma once
#include <fmt/format.h>
-#include <gen_cpp/Status_types.h> // for TStatus
-#include <gen_cpp/types.pb.h>
#include <glog/logging.h>
#include <cstdint>
@@ -28,57 +26,59 @@ namespace doris {
class Status;
class PStatus;
+class TStatus;
namespace ErrorCode {
-// E thrift_error_name, print_stacktrace
-#define APPLY_FOR_THRIFT_ERROR_CODES(TStatusError) \
- TStatusError(PUBLISH_TIMEOUT, false); \
- TStatusError(MEM_ALLOC_FAILED, true); \
- TStatusError(BUFFER_ALLOCATION_FAILED, true); \
- TStatusError(INVALID_ARGUMENT, false); \
- TStatusError(INVALID_JSON_PATH, false); \
- TStatusError(MINIMUM_RESERVATION_UNAVAILABLE, true); \
- TStatusError(CORRUPTION, true); \
- TStatusError(IO_ERROR, true); \
- TStatusError(NOT_FOUND, true); \
- TStatusError(ALREADY_EXIST, true); \
- TStatusError(DIRECTORY_NOT_EMPTY, true); \
- TStatusError(NOT_IMPLEMENTED_ERROR, false); \
- TStatusError(END_OF_FILE, false); \
- TStatusError(INTERNAL_ERROR, true); \
- TStatusError(RUNTIME_ERROR, true); \
- TStatusError(JNI_ERROR, true); \
- TStatusError(CANCELLED, false); \
- TStatusError(ANALYSIS_ERROR, false); \
- TStatusError(MEM_LIMIT_EXCEEDED, false); \
- TStatusError(THRIFT_RPC_ERROR, true); \
- TStatusError(TIMEOUT, true); \
- TStatusError(LIMIT_REACH, false); \
- TStatusError(TOO_MANY_TASKS, true); \
- TStatusError(UNINITIALIZED, false); \
- TStatusError(INCOMPLETE, false); \
- TStatusError(OLAP_ERR_VERSION_ALREADY_MERGED, false); \
- TStatusError(ABORTED, false); \
- TStatusError(DATA_QUALITY_ERROR, false); \
- TStatusError(LABEL_ALREADY_EXISTS, true); \
- TStatusError(NOT_AUTHORIZED, true); \
- TStatusError(BINLOG_DISABLE, false); \
- TStatusError(BINLOG_TOO_OLD_COMMIT_SEQ, false); \
- TStatusError(BINLOG_TOO_NEW_COMMIT_SEQ, false); \
- TStatusError(BINLOG_NOT_FOUND_DB, false); \
- TStatusError(BINLOG_NOT_FOUND_TABLE, false); \
- TStatusError(NETWORK_ERROR, false); \
- TStatusError(ILLEGAL_STATE, false); \
- TStatusError(SNAPSHOT_NOT_EXIST, true); \
- TStatusError(HTTP_ERROR, true); \
- TStatusError(TABLET_MISSING, true); \
- TStatusError(NOT_MASTER, true); \
- TStatusError(OBTAIN_LOCK_FAILED, false); \
- TStatusError(SNAPSHOT_EXPIRED, false); \
- TStatusError(DELETE_BITMAP_LOCK_ERROR, false); \
- TStatusError(SC_COMPACTION_CONFLICT, false); \
- TStatusError(FINISHED, false);
+// E thrift_error_name, thrift_error_code (must match Status.thrift; the
pairing
+// is enforced by static_asserts in status.cpp), print_stacktrace
+#define APPLY_FOR_THRIFT_ERROR_CODES(TStatusError) \
+ TStatusError(PUBLISH_TIMEOUT, 14, false); \
+ TStatusError(MEM_ALLOC_FAILED, 11, true); \
+ TStatusError(BUFFER_ALLOCATION_FAILED, 12, true); \
+ TStatusError(INVALID_ARGUMENT, 33, false); \
+ TStatusError(INVALID_JSON_PATH, 47, false); \
+ TStatusError(MINIMUM_RESERVATION_UNAVAILABLE, 13, true); \
+ TStatusError(CORRUPTION, 32, true); \
+ TStatusError(IO_ERROR, 34, true); \
+ TStatusError(NOT_FOUND, 31, true); \
+ TStatusError(ALREADY_EXIST, 35, true); \
+ TStatusError(DIRECTORY_NOT_EMPTY, 40, true); \
+ TStatusError(NOT_IMPLEMENTED_ERROR, 3, false); \
+ TStatusError(END_OF_FILE, 30, false); \
+ TStatusError(INTERNAL_ERROR, 6, true); \
+ TStatusError(RUNTIME_ERROR, 4, true); \
+ TStatusError(JNI_ERROR, 48, true); \
+ TStatusError(CANCELLED, 1, false); \
+ TStatusError(ANALYSIS_ERROR, 2, false); \
+ TStatusError(MEM_LIMIT_EXCEEDED, 5, false); \
+ TStatusError(THRIFT_RPC_ERROR, 7, true); \
+ TStatusError(TIMEOUT, 8, true); \
+ TStatusError(LIMIT_REACH, 9, false); \
+ TStatusError(TOO_MANY_TASKS, 16, true); \
+ TStatusError(UNINITIALIZED, 42, false); \
+ TStatusError(INCOMPLETE, 44, false); \
+ TStatusError(OLAP_ERR_VERSION_ALREADY_MERGED, 45, false); \
+ TStatusError(ABORTED, 39, false); \
+ TStatusError(DATA_QUALITY_ERROR, 46, false); \
+ TStatusError(LABEL_ALREADY_EXISTS, 15, true); \
+ TStatusError(NOT_AUTHORIZED, 38, true); \
+ TStatusError(BINLOG_DISABLE, 60, false); \
+ TStatusError(BINLOG_TOO_OLD_COMMIT_SEQ, 61, false); \
+ TStatusError(BINLOG_TOO_NEW_COMMIT_SEQ, 62, false); \
+ TStatusError(BINLOG_NOT_FOUND_DB, 63, false); \
+ TStatusError(BINLOG_NOT_FOUND_TABLE, 64, false); \
+ TStatusError(NETWORK_ERROR, 36, false); \
+ TStatusError(ILLEGAL_STATE, 37, false); \
+ TStatusError(SNAPSHOT_NOT_EXIST, 70, true); \
+ TStatusError(HTTP_ERROR, 71, true); \
+ TStatusError(TABLET_MISSING, 72, true); \
+ TStatusError(NOT_MASTER, 73, true); \
+ TStatusError(OBTAIN_LOCK_FAILED, 74, false); \
+ TStatusError(SNAPSHOT_EXPIRED, 75, false); \
+ TStatusError(DELETE_BITMAP_LOCK_ERROR, 100, false); \
+ TStatusError(SC_COMPACTION_CONFLICT, 101, false); \
+ TStatusError(FINISHED, 76, false);
// E error_name, error_code, print_stacktrace
#define APPLY_FOR_OLAP_ERROR_CODES(E) \
E(OK, 0, false); \
@@ -308,7 +308,7 @@ namespace ErrorCode {
APPLY_FOR_OLAP_ERROR_CODES(M)
#undef M
-#define MM(name, ENABLESTACKTRACE) constexpr int name = TStatusCode::name;
+#define MM(name, value, ENABLESTACKTRACE) constexpr int name = (value);
APPLY_FOR_THRIFT_ERROR_CODES(MM)
#undef MM
@@ -328,10 +328,10 @@ public:
for (auto& error_state : error_states) {
error_state.error_code = 0;
}
-#define M(NAME, ENABLESTACKTRACE) \
- error_states[TStatusCode::NAME].stacktrace = ENABLESTACKTRACE; \
- error_states[TStatusCode::NAME].description = #NAME; \
- error_states[TStatusCode::NAME].error_code = TStatusCode::NAME;
+#define M(NAME, VALUE, ENABLESTACKTRACE) \
+ error_states[NAME].stacktrace = ENABLESTACKTRACE; \
+ error_states[NAME].description = #NAME; \
+ error_states[NAME].error_code = NAME;
APPLY_FOR_THRIFT_ERROR_CODES(M)
#undef M
// In status.h, if error code > 0, then it means it will be used in
TStatusCode and will
@@ -408,19 +408,13 @@ public:
return *this;
}
+ // Defined in status.cpp (explicitly instantiated for stacktrace =
true/false)
+ // so this header does not need the thrift/protobuf generated headers.
template <bool stacktrace = true>
- Status static create(const TStatus& status) {
- return Error<stacktrace>(
- status.status_code,
- "TStatus: " + (status.error_msgs.empty() ? "" :
status.error_msgs[0]));
- }
+ Status static create(const TStatus& status);
template <bool stacktrace = true>
- Status static create(const PStatus& pstatus) {
- return Error<stacktrace>(
- pstatus.status_code(),
- "PStatus: " + (pstatus.error_msgs_size() == 0 ? "" :
pstatus.error_msgs(0)));
- }
+ Status static create(const PStatus& pstatus);
template <int code, bool stacktrace = true, typename... Args>
Status static Error(std::string_view msg, Args&&... args) {
@@ -580,10 +574,7 @@ private:
};
std::unique_ptr<ErrMsg> _err_msg;
- std::string code_as_string() const {
- return (int)_code >= 0 ?
doris::to_string(static_cast<TStatusCode::type>(_code))
- : fmt::format("E{}", (int16_t)_code);
- }
+ std::string code_as_string() const;
};
// There are many thread using status to indicate the cancel state, one thread
may update it and
diff --git a/be/src/core/data_type/data_type.cpp
b/be/src/core/data_type/data_type.cpp
index 941a0377b05..ce7d6b627f1 100644
--- a/be/src/core/data_type/data_type.cpp
+++ b/be/src/core/data_type/data_type.cpp
@@ -80,6 +80,14 @@ void IDataType::to_pb_column_meta(PColumnMeta* col_meta)
const {
col_meta->set_type(get_pdata_type(this));
}
+void IDataType::to_protobuf(PTypeDesc* ptype) const {
+ auto node = ptype->add_types();
+ node->set_type(TTypeNodeType::SCALAR);
+ auto scalar_type = node->mutable_scalar_type();
+ scalar_type->set_type(doris::to_thrift(get_primitive_type()));
+ to_protobuf(ptype, node, scalar_type);
+}
+
PGenericType_TypeId IDataType::get_pdata_type(const IDataType* data_type) {
switch (data_type->get_primitive_type()) {
case PrimitiveType::TYPE_BOOLEAN:
diff --git a/be/src/core/data_type/data_type.h
b/be/src/core/data_type/data_type.h
index e1042bf4161..f0c6ad80596 100644
--- a/be/src/core/data_type/data_type.h
+++ b/be/src/core/data_type/data_type.h
@@ -44,6 +44,9 @@
namespace doris {
class PColumnMeta;
+class PTypeDesc;
+class PTypeNode;
+class PScalarType;
enum PGenericType_TypeId : int;
class IDataType;
@@ -165,13 +168,9 @@ public:
[[nodiscard]] virtual UInt32 get_precision() const { return 0; }
[[nodiscard]] virtual UInt32 get_scale() const { return 0; }
virtual void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const {}
- void to_protobuf(PTypeDesc* ptype) const {
- auto node = ptype->add_types();
- node->set_type(TTypeNodeType::SCALAR);
- auto scalar_type = node->mutable_scalar_type();
- scalar_type->set_type(doris::to_thrift(get_primitive_type()));
- to_protobuf(ptype, node, scalar_type);
- }
+ // Defined in data_type.cpp: the body dereferences the protobuf types, and
+ // keeping it out of line keeps gen_cpp/types.pb.h out of this header.
+ void to_protobuf(PTypeDesc* ptype) const;
#ifdef BE_TEST
// only used in beut
Status from_string(StringRef& str, IColumn* column) const {
diff --git a/be/src/core/data_type/data_type_array.cpp
b/be/src/core/data_type/data_type_array.cpp
index 1cd003e4289..3ac5bd68dd1 100644
--- a/be/src/core/data_type/data_type_array.cpp
+++ b/be/src/core/data_type/data_type_array.cpp
@@ -22,6 +22,7 @@
#include <ctype.h>
#include <gen_cpp/data.pb.h>
+#include <gen_cpp/types.pb.h>
#include <glog/logging.h>
#include <string.h>
@@ -194,4 +195,10 @@ FieldWithDataType
DataTypeArray::get_field_with_data_type(const IColumn& column,
};
}
+void DataTypeArray::to_protobuf(PTypeDesc* ptype, PTypeNode* node,
PScalarType* scalar_type) const {
+ node->set_type(TTypeNodeType::ARRAY);
+ node->set_contains_null(nested->is_nullable());
+ get_nested_type()->to_protobuf(ptype);
+}
+
} // namespace doris
diff --git a/be/src/core/data_type/data_type_array.h
b/be/src/core/data_type/data_type_array.h
index 26479859121..5dc3623387a 100644
--- a/be/src/core/data_type/data_type_array.h
+++ b/be/src/core/data_type/data_type_array.h
@@ -94,11 +94,7 @@ public:
return std::make_shared<SerDeType>(nested->get_serde(nesting_level +
1), nesting_level);
};
- void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override {
- node->set_type(TTypeNodeType::ARRAY);
- node->set_contains_null(nested->is_nullable());
- get_nested_type()->to_protobuf(ptype);
- }
+ void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override;
#ifdef BE_TEST
void to_thrift(TTypeDesc& thrift_type, TTypeNode& node) const override {
diff --git a/be/src/core/data_type/data_type_date_or_datetime_v2.cpp
b/be/src/core/data_type/data_type_date_or_datetime_v2.cpp
index 35b3ba7c56a..ea01c3e300b 100644
--- a/be/src/core/data_type/data_type_date_or_datetime_v2.cpp
+++ b/be/src/core/data_type/data_type_date_or_datetime_v2.cpp
@@ -18,6 +18,7 @@
#include "core/data_type/data_type_date_or_datetime_v2.h"
#include <gen_cpp/data.pb.h>
+#include <gen_cpp/types.pb.h>
#include <memory>
#include <ostream>
@@ -191,6 +192,11 @@ DataTypePtr create_datetimev2(UInt64 scale_value) {
return std::make_shared<DataTypeDateTimeV2>(scale_value);
}
+void DataTypeDateTimeV2::to_protobuf(PTypeDesc* ptype, PTypeNode* node,
+ PScalarType* scalar_type) const {
+ scalar_type->set_scale(_scale);
+}
+
} // namespace doris
#if defined(__GNUC__) && (__GNUC__ >= 15)
diff --git a/be/src/core/data_type/data_type_date_or_datetime_v2.h
b/be/src/core/data_type/data_type_date_or_datetime_v2.h
index 0e16644863e..f36ce3a5149 100644
--- a/be/src/core/data_type/data_type_date_or_datetime_v2.h
+++ b/be/src/core/data_type/data_type_date_or_datetime_v2.h
@@ -104,9 +104,7 @@ public:
DataTypeDateTimeV2(const DataTypeDateTimeV2& rhs) : _scale(rhs._scale) {}
PrimitiveType get_primitive_type() const override { return
PrimitiveType::TYPE_DATETIMEV2; }
- void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override {
- scalar_type->set_scale(_scale);
- }
+ void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override;
const std::string get_family_name() const override { return "DateTimeV2"; }
std::string do_get_name() const override {
diff --git a/be/src/core/data_type/data_type_decimal.cpp
b/be/src/core/data_type/data_type_decimal.cpp
index 752275d9580..24b56d5b142 100644
--- a/be/src/core/data_type/data_type_decimal.cpp
+++ b/be/src/core/data_type/data_type_decimal.cpp
@@ -22,6 +22,7 @@
#include <fmt/format.h>
#include <gen_cpp/data.pb.h>
+#include <gen_cpp/types.pb.h>
#include <streamvbyte.h>
#include <sys/types.h>
@@ -274,6 +275,14 @@ FieldWithDataType
DataTypeDecimal<T>::get_field_with_data_type(const IColumn& co
}
/// Explicit template instantiations.
+
+template <PrimitiveType T>
+void DataTypeDecimal<T>::to_protobuf(PTypeDesc* ptype, PTypeNode* node,
+ PScalarType* scalar_type) const {
+ scalar_type->set_precision(precision);
+ scalar_type->set_scale(scale);
+}
+
template class DataTypeDecimal<TYPE_DECIMAL32>;
template class DataTypeDecimal<TYPE_DECIMAL64>;
template class DataTypeDecimal<TYPE_DECIMALV2>;
diff --git a/be/src/core/data_type/data_type_decimal.h
b/be/src/core/data_type/data_type_decimal.h
index 7476c4d7bbd..3ab1396648d 100644
--- a/be/src/core/data_type/data_type_decimal.h
+++ b/be/src/core/data_type/data_type_decimal.h
@@ -308,10 +308,7 @@ public:
typename FieldType::NativeType get_scale_multiplier() const {
return get_scale_multiplier(scale);
}
- void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override {
- scalar_type->set_precision(precision);
- scalar_type->set_scale(scale);
- }
+ void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override;
/// @returns multiplier for U to become T with correct scale
template <PrimitiveType U>
diff --git a/be/src/core/data_type/data_type_map.cpp
b/be/src/core/data_type/data_type_map.cpp
index e57950c5ad5..9a36a355d5a 100644
--- a/be/src/core/data_type/data_type_map.cpp
+++ b/be/src/core/data_type/data_type_map.cpp
@@ -19,6 +19,7 @@
#include <ctype.h>
#include <gen_cpp/data.pb.h>
+#include <gen_cpp/types.pb.h>
#include <glog/logging.h>
#include <string.h>
@@ -148,4 +149,13 @@ const char* DataTypeMap::deserialize(const char* buf,
MutableColumnPtr* column,
map_column->get_values_ptr() = std::move(nested_values_column);
return buf;
}
+
+void DataTypeMap::to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const {
+ node->set_type(TTypeNodeType::MAP);
+ node->add_contains_nulls(key_type->is_nullable());
+ node->add_contains_nulls(value_type->is_nullable());
+ key_type->to_protobuf(ptype);
+ value_type->to_protobuf(ptype);
+}
+
} // namespace doris
diff --git a/be/src/core/data_type/data_type_map.h
b/be/src/core/data_type/data_type_map.h
index b442b64da9b..6338ca87029 100644
--- a/be/src/core/data_type/data_type_map.h
+++ b/be/src/core/data_type/data_type_map.h
@@ -85,13 +85,7 @@ public:
return std::make_shared<SerDeType>(key_type->get_serde(nesting_level +
1),
value_type->get_serde(nesting_level
+ 1), nesting_level);
};
- void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override {
- node->set_type(TTypeNodeType::MAP);
- node->add_contains_nulls(key_type->is_nullable());
- node->add_contains_nulls(value_type->is_nullable());
- key_type->to_protobuf(ptype);
- value_type->to_protobuf(ptype);
- }
+ void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override;
#ifdef BE_TEST
void to_thrift(TTypeDesc& thrift_type, TTypeNode& node) const override {
node.type = TTypeNodeType::MAP;
diff --git a/be/src/core/data_type/data_type_string.cpp
b/be/src/core/data_type/data_type_string.cpp
index 2fa6d3e99d8..bf53aa5c4d0 100644
--- a/be/src/core/data_type/data_type_string.cpp
+++ b/be/src/core/data_type/data_type_string.cpp
@@ -20,6 +20,7 @@
#include "core/data_type/data_type_string.h"
+#include <gen_cpp/types.pb.h>
#include <lz4/lz4.h>
#include <streamvbyte.h>
@@ -187,4 +188,11 @@ FieldWithDataType
DataTypeString::get_field_with_data_type(const IColumn& column
.base_scalar_type_id = get_primitive_type()};
}
+void DataTypeString::to_protobuf(PTypeDesc* ptype, PTypeNode* node,
+ PScalarType* scalar_type) const {
+ if (_primitive_type == TYPE_CHAR || _primitive_type == TYPE_VARCHAR) {
+ scalar_type->set_len(_len);
+ }
+}
+
} // namespace doris
diff --git a/be/src/core/data_type/data_type_string.h
b/be/src/core/data_type/data_type_string.h
index bfd86f7d9de..77eae7b6dee 100644
--- a/be/src/core/data_type/data_type_string.h
+++ b/be/src/core/data_type/data_type_string.h
@@ -79,11 +79,7 @@ public:
};
bool is_char_type() const { return _primitive_type ==
PrimitiveType::TYPE_CHAR; }
int len() const { return _len; }
- void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override {
- if (_primitive_type == TYPE_CHAR || _primitive_type == TYPE_VARCHAR) {
- scalar_type->set_len(_len);
- }
- }
+ void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override;
#ifdef BE_TEST
void to_thrift(TTypeDesc& thrift_type, TTypeNode& node) const override {
IDataType::to_thrift(thrift_type, node);
diff --git a/be/src/core/data_type/data_type_struct.cpp
b/be/src/core/data_type/data_type_struct.cpp
index 4b3d5291c31..76693087eae 100644
--- a/be/src/core/data_type/data_type_struct.cpp
+++ b/be/src/core/data_type/data_type_struct.cpp
@@ -23,6 +23,7 @@
#include <ctype.h>
#include <fmt/format.h>
#include <gen_cpp/data.pb.h>
+#include <gen_cpp/types.pb.h>
#include <glog/logging.h>
#include <string.h>
@@ -247,4 +248,17 @@ size_t DataTypeStruct::get_size_of_value_in_memory() const
{
return res;
}
+void DataTypeStruct::to_protobuf(PTypeDesc* ptype, PTypeNode* node,
+ PScalarType* scalar_type) const {
+ node->set_type(TTypeNodeType::STRUCT);
+ for (size_t i = 0; i < elems.size(); ++i) {
+ auto field = node->add_struct_fields();
+ field->set_name(get_element_name(i));
+ field->set_contains_null(elems[i]->is_nullable());
+ }
+ for (const auto& child : elems) {
+ child->to_protobuf(ptype);
+ }
+}
+
} // namespace doris
diff --git a/be/src/core/data_type/data_type_struct.h
b/be/src/core/data_type/data_type_struct.h
index d4e4abd7a88..5ef95b3f367 100644
--- a/be/src/core/data_type/data_type_struct.h
+++ b/be/src/core/data_type/data_type_struct.h
@@ -112,17 +112,7 @@ public:
}
return std::make_shared<SerDeType>(ptrs, names, nesting_level);
};
- void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override {
- node->set_type(TTypeNodeType::STRUCT);
- for (size_t i = 0; i < elems.size(); ++i) {
- auto field = node->add_struct_fields();
- field->set_name(get_element_name(i));
- field->set_contains_null(elems[i]->is_nullable());
- }
- for (const auto& child : elems) {
- child->to_protobuf(ptype);
- }
- }
+ void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override;
#ifdef BE_TEST
void to_thrift(TTypeDesc& thrift_type, TTypeNode& node) const override {
node.type = TTypeNodeType::STRUCT;
diff --git a/be/src/core/data_type/data_type_timestamptz.cpp
b/be/src/core/data_type/data_type_timestamptz.cpp
index 7c6b4d249cb..92d4473df70 100644
--- a/be/src/core/data_type/data_type_timestamptz.cpp
+++ b/be/src/core/data_type/data_type_timestamptz.cpp
@@ -21,6 +21,7 @@
#include "core/data_type/data_type_timestamptz.h"
#include <gen_cpp/data.pb.h>
+#include <gen_cpp/types.pb.h>
#include "exprs/function/cast/cast_to_timestamptz.h"
@@ -45,4 +46,9 @@ Field DataTypeTimeStampTz::get_field(const TExprNode& node)
const {
}
}
+void DataTypeTimeStampTz::to_protobuf(PTypeDesc* ptype, PTypeNode* node,
+ PScalarType* scalar_type) const {
+ scalar_type->set_scale(_scale);
+}
+
} // namespace doris
\ No newline at end of file
diff --git a/be/src/core/data_type/data_type_timestamptz.h
b/be/src/core/data_type/data_type_timestamptz.h
index 89b405e87b5..e2d5e590fed 100644
--- a/be/src/core/data_type/data_type_timestamptz.h
+++ b/be/src/core/data_type/data_type_timestamptz.h
@@ -56,9 +56,7 @@ public:
return "TimeStampTz(" + std::to_string(_scale) + ")";
}
- void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override {
- scalar_type->set_scale(_scale);
- }
+ void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override;
void to_pb_column_meta(PColumnMeta* col_meta) const override;
diff --git a/be/src/core/data_type/data_type_varbinary.cpp
b/be/src/core/data_type/data_type_varbinary.cpp
index 7fc31fe54f9..fd4364592a9 100644
--- a/be/src/core/data_type/data_type_varbinary.cpp
+++ b/be/src/core/data_type/data_type_varbinary.cpp
@@ -17,6 +17,7 @@
#include "core/data_type/data_type_varbinary.h"
+#include <gen_cpp/types.pb.h>
#include <glog/logging.h>
#include <lz4/lz4.h>
#include <streamvbyte.h>
@@ -128,4 +129,9 @@ FieldWithDataType
DataTypeVarbinary::get_field_with_data_type(const IColumn& col
.base_scalar_type_id = get_primitive_type()};
}
+void DataTypeVarbinary::to_protobuf(PTypeDesc* ptype, PTypeNode* node,
+ PScalarType* scalar_type) const {
+ scalar_type->set_len(_len);
+}
+
} // namespace doris
diff --git a/be/src/core/data_type/data_type_varbinary.h
b/be/src/core/data_type/data_type_varbinary.h
index 63b4ae4af2a..aa54a8a79d4 100644
--- a/be/src/core/data_type/data_type_varbinary.h
+++ b/be/src/core/data_type/data_type_varbinary.h
@@ -82,9 +82,7 @@ public:
return std::make_shared<DataTypeVarbinarySerDe>(nesting_level);
};
- void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override {
- scalar_type->set_len(_len);
- }
+ void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override;
int len() const { return _len; }
diff --git a/be/src/core/data_type/data_type_variant.cpp
b/be/src/core/data_type/data_type_variant.cpp
index fff6ff53408..231b44df623 100644
--- a/be/src/core/data_type/data_type_variant.cpp
+++ b/be/src/core/data_type/data_type_variant.cpp
@@ -21,6 +21,7 @@
#include "core/data_type/data_type_variant.h"
#include <gen_cpp/data.pb.h>
+#include <gen_cpp/types.pb.h>
#include <string.h>
#include <cassert>
@@ -249,4 +250,11 @@ MutableColumnPtr DataTypeVariant::create_column() const {
return ColumnVariant::create(_max_subcolumns_count, _enable_doc_mode);
}
+void DataTypeVariant::to_protobuf(PTypeDesc* ptype, PTypeNode* node,
+ PScalarType* scalar_type) const {
+ node->set_type(TTypeNodeType::VARIANT);
+ node->set_variant_max_subcolumns_count(_max_subcolumns_count);
+ node->set_variant_enable_doc_mode(_enable_doc_mode);
+}
+
} // namespace doris
diff --git a/be/src/core/data_type/data_type_variant.h
b/be/src/core/data_type/data_type_variant.h
index 3f4e08a0ada..6f5d066a3cb 100644
--- a/be/src/core/data_type/data_type_variant.h
+++ b/be/src/core/data_type/data_type_variant.h
@@ -77,11 +77,7 @@ public:
DataTypeSerDeSPtr get_serde(int nesting_level = 1) const override {
return std::make_shared<SerDeType>(nesting_level);
};
- void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override {
- node->set_type(TTypeNodeType::VARIANT);
- node->set_variant_max_subcolumns_count(_max_subcolumns_count);
- node->set_variant_enable_doc_mode(_enable_doc_mode);
- }
+ void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override;
void to_pb_column_meta(PColumnMeta* col_meta) const override;
int32_t variant_max_subcolumns_count() const { return
_max_subcolumns_count; }
bool enable_doc_mode() const { return _enable_doc_mode; }
diff --git a/be/src/core/data_type_serde/data_type_array_serde.cpp
b/be/src/core/data_type_serde/data_type_array_serde.cpp
index 32f3e044f5d..3df4d073cbb 100644
--- a/be/src/core/data_type_serde/data_type_array_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_array_serde.cpp
@@ -18,6 +18,7 @@
#include "core/data_type_serde/data_type_array_serde.h"
#include <arrow/array/builder_nested.h>
+#include <gen_cpp/types.pb.h>
#include <algorithm>
diff --git a/be/src/core/data_type_serde/data_type_map_serde.cpp
b/be/src/core/data_type_serde/data_type_map_serde.cpp
index 419006257a5..2702e14373d 100644
--- a/be/src/core/data_type_serde/data_type_map_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_map_serde.cpp
@@ -17,6 +17,8 @@
#include "core/data_type_serde/data_type_map_serde.h"
+#include <gen_cpp/types.pb.h>
+
#include <algorithm>
#include "arrow/array/builder_nested.h"
diff --git a/be/src/core/data_type_serde/data_type_struct_serde.cpp
b/be/src/core/data_type_serde/data_type_struct_serde.cpp
index 8d7cf1fa30e..cf13c11dcaf 100644
--- a/be/src/core/data_type_serde/data_type_struct_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_struct_serde.cpp
@@ -17,6 +17,8 @@
#include "core/data_type_serde/data_type_struct_serde.h"
+#include <gen_cpp/types.pb.h>
+
#include <algorithm>
#include "arrow/array/builder_nested.h"
diff --git a/be/src/exec/common/endian.h b/be/src/exec/common/endian.h
index 0707b656d8b..d7b6d28d328 100644
--- a/be/src/exec/common/endian.h
+++ b/be/src/exec/common/endian.h
@@ -22,6 +22,18 @@
#include "util/unaligned.h"
namespace doris {
+// This header used to lean on the bswap_16/32/64 macros that protobuf's
+// stubs/port.h happened to leak into most TUs; use the compiler builtins
+// directly so it stands on its own natural include closure (redefining the
+// macros here would clash with protobuf's unguarded definitions).
+inline uint16_t gbswap_16(uint16_t host_int) {
+ return __builtin_bswap16(host_int);
+}
+
+inline uint32_t gbswap_32(uint32_t host_int) {
+ return __builtin_bswap32(host_int);
+}
+
inline uint64_t gbswap_64(uint64_t host_int) {
#if defined(__GNUC__) && defined(__x86_64__) && !defined(__APPLE__)
// Adapted from /usr/include/byteswap.h. Not available on Mac.
@@ -32,17 +44,14 @@ inline uint64_t gbswap_64(uint64_t host_int) {
__asm__("bswap %0" : "=r"(result) : "0"(host_int));
return result;
}
-#elif defined(bswap_64)
- return bswap_64(host_int);
#else
- return static_cast<uint64_t>(bswap_32(static_cast<uint32_t>(host_int >>
32))) |
- (static_cast<uint64_t>(bswap_32(static_cast<uint32_t>(host_int)))
<< 32);
-#endif // bswap_64
+ return __builtin_bswap64(host_int);
+#endif
}
inline unsigned __int128 gbswap_128(unsigned __int128 host_int) {
- return static_cast<unsigned
__int128>(bswap_64(static_cast<uint64_t>(host_int >> 64))) |
- (static_cast<unsigned
__int128>(bswap_64(static_cast<uint64_t>(host_int))) << 64);
+ return static_cast<unsigned
__int128>(gbswap_64(static_cast<uint64_t>(host_int >> 64))) |
+ (static_cast<unsigned
__int128>(gbswap_64(static_cast<uint64_t>(host_int))) << 64);
}
inline wide::UInt256 gbswap_256(wide::UInt256 host_int) {
@@ -64,13 +73,13 @@ T byte_swap(T x) {
} else if constexpr (sizeof(T) == sizeof(__int128)) {
return gbswap_128(x);
} else if constexpr (sizeof(T) == sizeof(int64_t)) {
- return bswap_64(x);
+ return gbswap_64(x);
} else if constexpr (sizeof(T) == sizeof(int32_t)) {
- return bswap_32(x);
+ return gbswap_32(x);
} else if constexpr (sizeof(T) == sizeof(doris::uint24_t)) {
return bswap_24(x);
} else if constexpr (sizeof(T) == sizeof(int16_t)) {
- return bswap_16(x);
+ return gbswap_16(x);
} else {
static_assert(sizeof(T) == 1, "Unsupported type size for byte_swap");
return x; // No byte swap needed for unsupported types
diff --git a/be/src/exec/scan/file_scanner.cpp
b/be/src/exec/scan/file_scanner.cpp
index b262cc6c900..d15b2c2b5f7 100644
--- a/be/src/exec/scan/file_scanner.cpp
+++ b/be/src/exec/scan/file_scanner.cpp
@@ -23,6 +23,7 @@
#include <gen_cpp/Opcodes_types.h>
#include <gen_cpp/PaloInternalService_types.h>
#include <gen_cpp/PlanNodes_types.h>
+#include <gen_cpp/Status_types.h>
#include <glog/logging.h>
#include <algorithm>
diff --git a/be/src/exec/scan/split_source_connector.cpp
b/be/src/exec/scan/split_source_connector.cpp
index 685a2d50f0c..1e22f8990d1 100644
--- a/be/src/exec/scan/split_source_connector.cpp
+++ b/be/src/exec/scan/split_source_connector.cpp
@@ -17,6 +17,8 @@
#include "exec/scan/split_source_connector.h"
+#include <gen_cpp/Status_types.h>
+
#include "runtime/exec_env.h"
#include "runtime/query_context.h"
diff --git a/be/src/exec/sink/vrow_distribution.cpp
b/be/src/exec/sink/vrow_distribution.cpp
index 5be03d22f53..f8f837e0695 100644
--- a/be/src/exec/sink/vrow_distribution.cpp
+++ b/be/src/exec/sink/vrow_distribution.cpp
@@ -19,6 +19,7 @@
#include <gen_cpp/FrontendService.h>
#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/Status_types.h>
#include <glog/logging.h>
#include <cstdint>
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index d78c9571e21..72dd4a6d7c7 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -30,6 +30,7 @@
#include <gen_cpp/Planner_types.h>
#include <gen_cpp/QueryPlanExtra_types.h>
#include <gen_cpp/RuntimeProfile_types.h>
+#include <gen_cpp/Status_types.h>
#include <gen_cpp/Types_types.h>
#include <gen_cpp/internal_service.pb.h>
#include <pthread.h>
diff --git a/be/src/service/backend_service.h b/be/src/service/backend_service.h
index 5f9c01f5ec0..b607790cc67 100644
--- a/be/src/service/backend_service.h
+++ b/be/src/service/backend_service.h
@@ -18,6 +18,7 @@
#pragma once
#include <gen_cpp/BackendService.h>
+#include <gen_cpp/Status_types.h>
#include <memory>
#include <string>
diff --git a/be/src/service/http/http_handler_with_auth.cpp
b/be/src/service/http/http_handler_with_auth.cpp
index d00734b7b74..dd3fdf9cfbc 100644
--- a/be/src/service/http/http_handler_with_auth.cpp
+++ b/be/src/service/http/http_handler_with_auth.cpp
@@ -18,6 +18,7 @@
#include "service/http/http_handler_with_auth.h"
#include <gen_cpp/HeartbeatService_types.h>
+#include <gen_cpp/Status_types.h>
#include "runtime/cluster_info.h"
#include "service/http/http_channel.h"
diff --git a/be/src/storage/index/inverted/setting.h
b/be/src/storage/index/inverted/setting.h
index 8e2a7ce1a72..87720fb6b27 100644
--- a/be/src/storage/index/inverted/setting.h
+++ b/be/src/storage/index/inverted/setting.h
@@ -25,6 +25,7 @@
#include <boost/algorithm/string/trim.hpp>
#include <boost/regex.hpp>
#include <unordered_map>
+#include <unordered_set>
#include <utility>
#include <vector>
diff --git a/be/src/storage/olap_server.cpp b/be/src/storage/olap_server.cpp
index 78021456ae2..2dc715cd327 100644
--- a/be/src/storage/olap_server.cpp
+++ b/be/src/storage/olap_server.cpp
@@ -26,6 +26,7 @@
#include <algorithm>
#include <atomic>
// IWYU pragma: no_include <bits/chrono.h>
+#include <gen_cpp/Status_types.h>
#include <gen_cpp/internal_service.pb.h>
#include <chrono> // IWYU pragma: keep
diff --git a/be/src/util/hash_util.hpp b/be/src/util/hash_util.hpp
index 49c01e4a176..ab5b72bffda 100644
--- a/be/src/util/hash_util.hpp
+++ b/be/src/util/hash_util.hpp
@@ -21,7 +21,6 @@
#pragma once
#include <crc32c/crc32c.h>
-#include <gen_cpp/Types_types.h>
#include <xxh3.h>
#include <xxhash.h>
#include <zlib.h>
@@ -469,36 +468,30 @@ public:
} // namespace doris
+namespace doris {
+// Forward declarations only: a std::hash specialization can be DECLARED for
+// an incomplete type, which keeps gen_cpp/Types_types.h out of this header.
+// This header stays the earliest carrier of these specializations (it rides
+// in through string_ref.h and storage/olap_common.h), so they are visible
+// before any implicit instantiation. Bodies live in util/uid_util.cpp and
+// util/network_util.cpp, where the types are complete.
+class TUniqueId;
+class TNetworkAddress;
+} // namespace doris
+
template <>
struct std::hash<doris::TUniqueId> {
- size_t operator()(const doris::TUniqueId& id) const {
- uint32_t seed = 0;
- seed = doris::HashUtil::hash(&id.lo, sizeof(id.lo), seed);
- seed = doris::HashUtil::hash(&id.hi, sizeof(id.hi), seed);
- return seed;
- }
+ size_t operator()(const doris::TUniqueId& id) const;
};
template <>
struct std::hash<doris::TNetworkAddress> {
- size_t operator()(const doris::TNetworkAddress& address) const {
- uint32_t seed = 0;
- seed = doris::HashUtil::hash(address.hostname.data(),
(uint32_t)address.hostname.size(),
- seed);
- seed = doris::HashUtil::hash(&address.port, 4, seed);
- return seed;
- }
+ size_t operator()(const doris::TNetworkAddress& address) const;
};
template <>
struct std::hash<std::pair<doris::TUniqueId, int64_t>> {
- size_t operator()(const std::pair<doris::TUniqueId, int64_t>& pair) const {
- uint32_t seed = 0;
- seed = doris::HashUtil::hash(&pair.first.lo, sizeof(pair.first.lo),
seed);
- seed = doris::HashUtil::hash(&pair.first.hi, sizeof(pair.first.hi),
seed);
- seed = doris::HashUtil::hash(&pair.second, sizeof(pair.second), seed);
- return seed;
- }
+ size_t operator()(const std::pair<doris::TUniqueId, int64_t>& pair) const;
};
template <class First, class Second>
diff --git a/be/src/util/network_util.cpp b/be/src/util/network_util.cpp
index ad1315dc121..968a7236661 100644
--- a/be/src/util/network_util.cpp
+++ b/be/src/util/network_util.cpp
@@ -38,6 +38,7 @@
#include <sstream>
#include "common/cast_set.h"
+#include "util/hash_util.hpp"
#ifdef __APPLE__
#ifndef HOST_NAME_MAX
@@ -305,3 +306,10 @@ std::string get_brpc_http_url(const std::string& host, int
port) {
}
}
} // namespace doris
+
+size_t std::hash<doris::TNetworkAddress>::operator()(const
doris::TNetworkAddress& address) const {
+ uint32_t seed = 0;
+ seed = doris::HashUtil::hash(address.hostname.data(),
(uint32_t)address.hostname.size(), seed);
+ seed = doris::HashUtil::hash(&address.port, 4, seed);
+ return seed;
+}
diff --git a/be/src/util/timezone_utils.cpp b/be/src/util/timezone_utils.cpp
index 358d8b962cc..e789ebd9e77 100644
--- a/be/src/util/timezone_utils.cpp
+++ b/be/src/util/timezone_utils.cpp
@@ -36,6 +36,7 @@
#include <cstdlib>
#include <filesystem>
#include <memory>
+#include <set>
#include <string>
#include <string_view>
diff --git a/be/src/util/uid_util.cpp b/be/src/util/uid_util.cpp
index 0ea26cfd4f0..4021bf68ac1 100644
--- a/be/src/util/uid_util.cpp
+++ b/be/src/util/uid_util.cpp
@@ -69,3 +69,19 @@ bool TUniqueId::operator<(const TUniqueId& rhs) const {
}
} // namespace doris
+
+size_t std::hash<doris::TUniqueId>::operator()(const doris::TUniqueId& id)
const {
+ uint32_t seed = 0;
+ seed = doris::HashUtil::hash(&id.lo, sizeof(id.lo), seed);
+ seed = doris::HashUtil::hash(&id.hi, sizeof(id.hi), seed);
+ return seed;
+}
+
+size_t std::hash<std::pair<doris::TUniqueId, int64_t>>::operator()(
+ const std::pair<doris::TUniqueId, int64_t>& pair) const {
+ uint32_t seed = 0;
+ seed = doris::HashUtil::hash(&pair.first.lo, sizeof(pair.first.lo), seed);
+ seed = doris::HashUtil::hash(&pair.first.hi, sizeof(pair.first.hi), seed);
+ seed = doris::HashUtil::hash(&pair.second, sizeof(pair.second), seed);
+ return seed;
+}
diff --git a/be/test/agent/heartbeat_server_test.cpp
b/be/test/agent/heartbeat_server_test.cpp
index 26e7fc5ccdc..38f8bda5103 100644
--- a/be/test/agent/heartbeat_server_test.cpp
+++ b/be/test/agent/heartbeat_server_test.cpp
@@ -18,6 +18,7 @@
#include "agent/heartbeat_server.h"
#include <gen_cpp/HeartbeatService_types.h>
+#include <gen_cpp/Status_types.h>
#include <gen_cpp/Types_types.h>
#include <ctime>
diff --git a/be/test/common/status_test.cpp b/be/test/common/status_test.cpp
index be555a0912c..afe76fbbb13 100644
--- a/be/test/common/status_test.cpp
+++ b/be/test/common/status_test.cpp
@@ -17,6 +17,7 @@
#include "common/status.h"
+#include <gen_cpp/Status_types.h>
#include <gtest/gtest-message.h>
#include <gtest/gtest-test-part.h>
diff --git
a/be/test/core/data_type_serde/data_type_serde_fixed_length_object_test.cpp
b/be/test/core/data_type_serde/data_type_serde_fixed_length_object_test.cpp
index d7c5473e921..a69760f69e5 100644
--- a/be/test/core/data_type_serde/data_type_serde_fixed_length_object_test.cpp
+++ b/be/test/core/data_type_serde/data_type_serde_fixed_length_object_test.cpp
@@ -16,6 +16,7 @@
// under the License.
#include <arrow/array/builder_base.h>
+#include <gen_cpp/types.pb.h>
#include <gtest/gtest.h>
#include "core/column/column_fixed_length_object.h"
diff --git a/be/test/exec/runtime_filter/sync_size_callback_test.cpp
b/be/test/exec/runtime_filter/sync_size_callback_test.cpp
index f208f54b05c..a088202090f 100644
--- a/be/test/exec/runtime_filter/sync_size_callback_test.cpp
+++ b/be/test/exec/runtime_filter/sync_size_callback_test.cpp
@@ -15,6 +15,7 @@
// specific language governing permissions and limitations
// under the License.
+#include <gen_cpp/Status_types.h>
#include <gtest/gtest.h>
#include "exec/runtime_filter/runtime_filter_producer.h"
diff --git a/be/test/exec/schema_scanner/schema_tso_status_scanner_test.cpp
b/be/test/exec/schema_scanner/schema_tso_status_scanner_test.cpp
index 67b82e2f181..60a95e3bffc 100644
--- a/be/test/exec/schema_scanner/schema_tso_status_scanner_test.cpp
+++ b/be/test/exec/schema_scanner/schema_tso_status_scanner_test.cpp
@@ -18,6 +18,7 @@
#include "information_schema/schema_tso_status_scanner.h"
#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/Status_types.h>
#include <gtest/gtest.h>
#include <array>
diff --git a/be/test/exec/sink/vrow_distribution_test.cpp
b/be/test/exec/sink/vrow_distribution_test.cpp
index 5d3110a7271..d0211a15f67 100644
--- a/be/test/exec/sink/vrow_distribution_test.cpp
+++ b/be/test/exec/sink/vrow_distribution_test.cpp
@@ -20,6 +20,7 @@
#include <gen_cpp/Exprs_types.h>
#include <gen_cpp/FrontendService_types.h>
#include <gen_cpp/Partitions_types.h>
+#include <gen_cpp/Status_types.h>
#include <gtest/gtest.h>
#include <cstdint>
diff --git a/be/test/io/cache/cached_remote_file_reader_peer_test.cpp
b/be/test/io/cache/cached_remote_file_reader_peer_test.cpp
index ebf96a59e27..0e8b79e4d1b 100644
--- a/be/test/io/cache/cached_remote_file_reader_peer_test.cpp
+++ b/be/test/io/cache/cached_remote_file_reader_peer_test.cpp
@@ -20,6 +20,7 @@
#include <butil/endpoint.h>
#include <butil/iobuf.h>
#include <bvar/bvar.h>
+#include <gen_cpp/Status_types.h>
#include <gen_cpp/internal_service.pb.h>
#include <algorithm>
diff --git a/be/test/io/fs/rate_limited_obj_storage_client_test.cpp
b/be/test/io/fs/rate_limited_obj_storage_client_test.cpp
index fce15d9ae75..a1c42c54557 100644
--- a/be/test/io/fs/rate_limited_obj_storage_client_test.cpp
+++ b/be/test/io/fs/rate_limited_obj_storage_client_test.cpp
@@ -17,6 +17,7 @@
#include "cpp/obj-client/rate_limited_obj_storage_client.h"
+#include <gen_cpp/Status_types.h>
#include <gtest/gtest.h>
#include <memory>
diff --git a/be/test/io/fs/s3_file_writer_test.cpp
b/be/test/io/fs/s3_file_writer_test.cpp
index 3f5d6670dba..b7bbd8330ee 100644
--- a/be/test/io/fs/s3_file_writer_test.cpp
+++ b/be/test/io/fs/s3_file_writer_test.cpp
@@ -26,6 +26,7 @@
#include <aws/s3/model/HeadObjectRequest.h>
#include <aws/s3/model/PutObjectRequest.h>
#include <aws/s3/model/UploadPartRequest.h>
+#include <gen_cpp/Status_types.h>
#include <gtest/gtest.h>
#include <any>
diff --git a/be/test/runtime/load_stream_test.cpp
b/be/test/runtime/load_stream_test.cpp
index b17f302d586..ce218d1ceff 100644
--- a/be/test/runtime/load_stream_test.cpp
+++ b/be/test/runtime/load_stream_test.cpp
@@ -21,8 +21,10 @@
#include <butil/logging.h>
#include <gen_cpp/BackendService_types.h>
#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/Status_types.h>
#include <gen_cpp/Types_types.h>
#include <gen_cpp/internal_service.pb.h>
+#include <gen_cpp/types.pb.h>
#include <gflags/gflags.h>
#include <gtest/gtest-message.h>
#include <gtest/gtest-test-part.h>
diff --git a/be/test/storage/segment/bloom_filter_index_reader_writer_test.cpp
b/be/test/storage/segment/bloom_filter_index_reader_writer_test.cpp
index 17eeac11a7d..3c6bb3db839 100644
--- a/be/test/storage/segment/bloom_filter_index_reader_writer_test.cpp
+++ b/be/test/storage/segment/bloom_filter_index_reader_writer_test.cpp
@@ -15,6 +15,7 @@
// specific language governing permissions and limitations
// under the License.
+#include <gen_cpp/Status_types.h>
#include <gen_cpp/segment_v2.pb.h>
#include <gtest/gtest-message.h>
#include <gtest/gtest-test-part.h>
diff --git a/be/test/storage/segment/bloom_filter_test.cpp
b/be/test/storage/segment/bloom_filter_test.cpp
index 27f36a6d43e..501dca02aa5 100644
--- a/be/test/storage/segment/bloom_filter_test.cpp
+++ b/be/test/storage/segment/bloom_filter_test.cpp
@@ -16,6 +16,7 @@
// under the License.
#include "storage/index/bloom_filter/bloom_filter.h"
+#include <gen_cpp/Status_types.h>
#include <gtest/gtest.h>
#include <iostream>
diff --git a/be/test/testutil/mock/obj_storage_client_test_stub.h
b/be/test/testutil/mock/obj_storage_client_test_stub.h
index 6cec949121a..2e02b2b6a93 100644
--- a/be/test/testutil/mock/obj_storage_client_test_stub.h
+++ b/be/test/testutil/mock/obj_storage_client_test_stub.h
@@ -17,6 +17,8 @@
#pragma once
+#include <gen_cpp/Status_types.h>
+
#include "cpp/obj-client/obj_storage_client.h"
namespace doris::io {
diff --git a/build-support/check-header-deps.py
b/build-support/check-header-deps.py
index 5c5783991e6..109f4b4d8d5 100755
--- a/build-support/check-header-deps.py
+++ b/build-support/check-header-deps.py
@@ -142,18 +142,42 @@ RULES = [
"options.h carries gen_cpp/Types_types.h and
io/cache/file_cache_common.h "
"into every TU that includes ExecEnv",
),
+ (
+ "common/status.h",
+ "gen_cpp/",
+ set(),
+ "Status is included by essentially every TU; its error-code constants "
+ "carry literal values pinned to Status.thrift by static_asserts in "
+ "status.cpp, and the TStatus/PStatus converters are declared on "
+ "forward declarations with bodies in status.cpp, precisely so no "
+ "generated thrift/protobuf header rides this superhighway",
+ ),
+ (
+ "common/exception.h",
+ "gen_cpp/",
+ set(),
+ "Exception only uses ErrorCode constants and Status; it reaches "
+ "nearly every TU through status/exception macros and must stay free "
+ "of generated headers for the same reason as common/status.h",
+ ),
+ (
+ "util/hash_util.hpp",
+ "gen_cpp/",
+ set(),
+ "hash_util is carried by string_ref.h, column_string.h, "
+ "vdatetime_value.h and storage/olap_common.h into most of the "
+ "backend; the std::hash specializations for TUniqueId and "
+ "TNetworkAddress live with their carriers (util/uid_util.h, "
+ "util/network_util.h), so no thrift header is needed here",
+ ),
(
"runtime/exec_env.h",
"gen_cpp/",
- {
- # Status embeds TStatus/PStatus; these two ride in through
- # common/status.h and are the only generated headers ExecEnv may
keep.
- "gen_cpp/Status_types.h",
- "gen_cpp/types.pb.h",
- },
+ set(),
"ExecEnv reaches ~1060 TUs, so any generated protobuf/thrift header it
"
"pulls in is reparsed by most of the backend; every thrift struct it "
- "stores is behind a pointer or forward declaration",
+ "stores is behind a pointer or forward declaration (the old "
+ "Status_types/types.pb ride-along died with the common/status.h cut)",
),
(
"runtime/thread_context.h",
@@ -194,10 +218,10 @@ RULES = [
"runtime/thread_context.h",
"gen_cpp/",
{
- # Status embeds TStatus/PStatus (ride in through common/status.h);
- # TUniqueId lives in Types_types.h; the profile family rides in
- # through runtime_profile.h held by the memory-tracker chain.
- "gen_cpp/Status_types.h",
+ # PUniqueId is embedded by value in util/uid_util.h (rides in via
+ # mem_tracker_limiter.h); TUniqueId lives in Types_types.h; the
+ # profile family rides in through runtime_profile.h held by the
+ # memory-tracker chain.
"gen_cpp/types.pb.h",
"gen_cpp/Types_types.h",
"gen_cpp/Metrics_types.h",
@@ -205,7 +229,7 @@ RULES = [
"gen_cpp/runtime_profile.pb.h",
},
"ThreadContext reaches ~1000 TUs; any generated protobuf/thrift header
"
- "beyond the status/types/profile carriers listed here is reparsed by "
+ "beyond the types/profile carriers listed here is reparsed by "
"most of the backend",
),
(
@@ -482,6 +506,13 @@ FORWARD_CLOSURE_BUDGETS = {
# intended.
# Baselines: master 9a48f8120c0, 2026-08-18.
REVERSE_REACH_BASELINES = {
+ # Locked down by the common/status.h decouple (P35): Status_types must not
+ # creep back toward its old everything-line reach of 2453 TUs.
+ "gen_cpp/Status_types.h": 37,
+ # Still carried legitimately by util/uid_util.h (PUniqueId by value) and
+ # the profile family; the status.h cut brought these down from 2458/2091.
+ "gen_cpp/types.pb.h": 1791,
+ "gen_cpp/Types_types.h": 1993,
"gen_cpp/segment_v2.pb.h": 1234,
"gen_cpp/PaloInternalService_types.h": 1133,
"util/threadpool.h": 988,
diff --git a/build-support/compile-bench/syntax_sweep.py
b/build-support/compile-bench/syntax_sweep.py
index e3f77bfa023..9918e5760ae 100644
--- a/build-support/compile-bench/syntax_sweep.py
+++ b/build-support/compile-bench/syntax_sweep.py
@@ -99,6 +99,9 @@ def mangle(cmd, no_pch=False):
def display_name(path, src_prefix):
if path.startswith(src_prefix):
return path[len(src_prefix):]
+ be_prefix = os.path.dirname(os.path.dirname(src_prefix)) + os.sep
+ if path.startswith(be_prefix + "test" + os.sep):
+ return path[len(be_prefix):]
if "/Unity/" in path:
# .../src/<dir>/CMakeFiles/<tgt>.dir/Unity/unity_N_cxx.cxx
m = re.search(r"/src/([^/]+)/CMakeFiles/[^/]+/Unity/(unity_\d+)", path)
@@ -107,12 +110,14 @@ def display_name(path, src_prefix):
return path
-def first_party(e, src_prefix):
+def first_party(e, src_prefix, test_prefix=None):
f = e["file"]
if f.startswith(src_prefix):
return True
+ if test_prefix and f.startswith(test_prefix):
+ return True
# CMake unity batches for first-party targets live under the build dir
- return "/Unity/" in f and "/src/" in f
+ return "/Unity/" in f and ("/src/" in f or (test_prefix and "/test/" in f))
def main():
@@ -131,12 +136,19 @@ def main():
help="write full stderr of every failing TU to this file")
ap.add_argument("--timeout", type=int, default=600,
help="per-TU timeout in seconds (counts as failure)")
+ ap.add_argument("--include-tests", action="store_true",
+ help="also sweep be/test TUs (point --build-dir at a "
+ "-DMAKE_TEST=ON tree; be/test builds without a PCH, "
+ "so the UT line always compiles natural closures)")
args = ap.parse_args()
src_prefix = os.path.join(REPO_ROOT, "be", "src") + os.sep
+ test_prefix = (os.path.join(REPO_ROOT, "be", "test") + os.sep
+ if args.include_tests else None)
with open(os.path.join(args.build_dir, "compile_commands.json")) as f:
entries = [e for e in json.load(f)
- if first_party(e, src_prefix) and args.filter in e["file"]]
+ if first_party(e, src_prefix, test_prefix)
+ and args.filter in e["file"]]
print(f"{len(entries)} TUs to check ({args.jobs} jobs"
f"{', natural closure / no PCH' if args.no_pch else ''})",
flush=True)
t0 = time.time()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
