This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 18d470ecf7 [improvement](config) add a specific be config for
segment_cache_capacity (#23701)
18d470ecf7 is described below
commit 18d470ecf7f2be17a1a672fdbe74d65be7871b37
Author: Kang <[email protected]>
AuthorDate: Sat Sep 2 01:14:14 2023 +0800
[improvement](config) add a specific be config for segment_cache_capacity
(#23701)
* add segment_cache_capacity config istead of fd limit * 2/5
* default -1 for backward compatibility
---
be/src/common/config.cpp | 3 +++
be/src/common/config.h | 3 +++
be/src/runtime/exec_env_init.cpp | 7 +++++--
docs/en/docs/admin-manual/config/be-config.md | 12 +++++-------
docs/zh-CN/docs/admin-manual/config/be-config.md | 12 +++++-------
5 files changed, 21 insertions(+), 16 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 558d35b522..67f06b743a 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1031,6 +1031,9 @@ DEFINE_Bool(enable_shrink_memory, "false");
DEFINE_mInt32(schema_cache_capacity, "1024");
DEFINE_mInt32(schema_cache_sweep_time_sec, "100");
+// max number of segment cache, default -1 for backward compatibility
fd_number*2/5
+DEFINE_mInt32(segment_cache_capacity, "-1");
+
// enable feature binlog, default false
DEFINE_Bool(enable_feature_binlog, "false");
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 5c098b0806..f2ef58ae0e 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1071,6 +1071,9 @@ DECLARE_Bool(enable_shrink_memory);
DECLARE_mInt32(schema_cache_capacity);
DECLARE_mInt32(schema_cache_sweep_time_sec);
+// max number of segment cache
+DECLARE_mInt32(segment_cache_capacity);
+
// enable binlog
DECLARE_Bool(enable_feature_binlog);
diff --git a/be/src/runtime/exec_env_init.cpp b/be/src/runtime/exec_env_init.cpp
index d3d4df8a34..5cf17d9e37 100644
--- a/be/src/runtime/exec_env_init.cpp
+++ b/be/src/runtime/exec_env_init.cpp
@@ -312,8 +312,11 @@ Status ExecEnv::_init_mem_env() {
}
// SegmentLoader caches segments in rowset granularity. So the size of
// opened files will greater than segment_cache_capacity.
- uint64_t segment_cache_capacity = fd_number * 2 / 5;
- LOG(INFO) << "segment_cache_capacity = fd_number * 2 / 5, fd_number: " <<
fd_number
+ int64_t segment_cache_capacity = config::segment_cache_capacity;
+ if (segment_cache_capacity < 0 || segment_cache_capacity > fd_number * 2 /
5) {
+ segment_cache_capacity = fd_number * 2 / 5;
+ }
+ LOG(INFO) << "segment_cache_capacity <= fd_number * 2 / 5, fd_number: " <<
fd_number
<< " segment_cache_capacity: " << segment_cache_capacity;
SegmentLoader::create_global_instance(segment_cache_capacity);
diff --git a/docs/en/docs/admin-manual/config/be-config.md
b/docs/en/docs/admin-manual/config/be-config.md
index e350d188cc..a22186cbf0 100644
--- a/docs/en/docs/admin-manual/config/be-config.md
+++ b/docs/en/docs/admin-manual/config/be-config.md
@@ -1028,13 +1028,6 @@ BaseCompaction:546859:
- Increasing this value can reduce the number of calls to read remote data,
but it will increase memory overhead.
* Default value: 16 (MB)
-#### `segment_cache_capacity`
-
-* Type: int32
-* Description: The maximum number of Segments cached by Segment Cache.
- - The default value is currently only an empirical value, and may need to be
modified according to actual scenarios. Increasing this value can cache more
segments and avoid some IO. Decreasing this value will reduce memory usage.
-* Default value: 1000000
-
#### `file_cache_type`
* Type: string
@@ -1197,6 +1190,11 @@ BaseCompaction:546859:
* Description: Index page cache as a percentage of total storage page cache,
value range is [0, 100]
* Default value: 10
+#### `segment_cache_capacity`
+* Type: int32
+* Description: Max number of segment cache (the key is rowset id) entries. -1
is for backward compatibility as fd_number * 2/5.
+* Default value: -1
+
#### `storage_strict_check_incompatible_old_format`
* Type: bool
diff --git a/docs/zh-CN/docs/admin-manual/config/be-config.md
b/docs/zh-CN/docs/admin-manual/config/be-config.md
index 3c7574d297..14eefd09d6 100644
--- a/docs/zh-CN/docs/admin-manual/config/be-config.md
+++ b/docs/zh-CN/docs/admin-manual/config/be-config.md
@@ -1053,13 +1053,6 @@ BaseCompaction:546859:
- 增大这个值，可以减少远端数据读取的调用次数，但会增加内存开销。
* 默认值: 16MB
-#### `segment_cache_capacity`
-
-* 类型: int32
-* 描述: Segment Cache 缓存的 Segment 最大数量
- - 默认值目前只是一个经验值，可能需要根据实际场景修改。增大该值可以缓存更多的segment从而避免一些IO。减少该值则会降低内存使用。
-* 默认值: 1000000
-
#### `file_cache_type`
* 类型：string
@@ -1222,6 +1215,11 @@ BaseCompaction:546859:
* 描述：索引页缓存占总页面缓存的百分比，取值为[0, 100]。
* 默认值：10
+#### `segment_cache_capacity`
+* Type: int32
+* Description: segment元数据缓存（以rowset id为key）的最大rowset个数. -1代表向后兼容取值为fd_number *
2/5
+* Default value: -1
+
#### `storage_strict_check_incompatible_old_format`
* 类型：bool
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
