Re: [PR] [improvement](spill) improve spill log printing [doris]
github-actions[bot] commented on PR #34729: URL: https://github.com/apache/doris/pull/34729#issuecomment-2106123421 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](nereids) support Iceberg time travel syntax [doris]
zhangbutao commented on PR #34681: URL: https://github.com/apache/doris/pull/34681#issuecomment-2106114670 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
doris-robot commented on PR #34679: URL: https://github.com/apache/doris/pull/34679#issuecomment-2106108677 TeamCity be ut coverage result: Function Coverage: 35.65% (8986/25205) Line Coverage: 27.31% (74261/271874) Region Coverage: 26.55% (38381/144560) Branch Coverage: 23.37% (19574/83764) Coverage Report: http://coverage.selectdb-in.cc/coverage/b58bc5fddaa6fd06b74625e6070a6b3be32750a9_b58bc5fddaa6fd06b74625e6070a6b3be32750a9/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ui) web profile download use plain text [doris]
yiguolei commented on code in PR #34705:
URL: https://github.com/apache/doris/pull/34705#discussion_r1597545240
##
fe/fe-core/src/main/java/org/apache/doris/httpv2/controller/QueryProfileController.java:
##
@@ -59,6 +59,15 @@ public Object profile(@PathVariable(value = ID) String id) {
return ResponseEntityBuilder.ok(profile);
}
+@RequestMapping(path = "/query_profile/text/{" + ID + "}", method =
RequestMethod.GET)
Review Comment:
when this API will be used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](memory) Fix prune all LRU Cache based on number #34601 (#34736)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 7a172a55aba [fix](memory) Fix prune all LRU Cache based on number
#34601 (#34736)
7a172a55aba is described below
commit 7a172a55aba96ff0d6a77871bfcc693a3c56af39
Author: Xinyi Zou
AuthorDate: Sun May 12 11:47:53 2024 +0800
[fix](memory) Fix prune all LRU Cache based on number #34601 (#34736)
---
be/src/runtime/memory/cache_policy.h | 1 +
be/src/runtime/memory/lru_cache_policy.h | 46 +++-
2 files changed, 34 insertions(+), 13 deletions(-)
diff --git a/be/src/runtime/memory/cache_policy.h
b/be/src/runtime/memory/cache_policy.h
index a7c421ed9a7..219ad2a27cd 100644
--- a/be/src/runtime/memory/cache_policy.h
+++ b/be/src/runtime/memory/cache_policy.h
@@ -24,6 +24,7 @@
namespace doris {
static constexpr int32_t CACHE_MIN_FREE_SIZE = 67108864; // 64M
+static constexpr int32_t CACHE_MIN_FREE_NUMBER = 1024;
// Base of all caches. register to CacheManager when cache is constructed.
class CachePolicy {
diff --git a/be/src/runtime/memory/lru_cache_policy.h
b/be/src/runtime/memory/lru_cache_policy.h
index 8f0806e9da9..773817393c7 100644
--- a/be/src/runtime/memory/lru_cache_policy.h
+++ b/be/src/runtime/memory/lru_cache_policy.h
@@ -124,7 +124,10 @@ public:
uint64_t new_id() { return _cache->new_id(); };
// Subclass can override this method to determine whether to do the minor
or full gc
-virtual bool exceed_prune_limit() { return mem_consumption() >
CACHE_MIN_FREE_SIZE; }
+virtual bool exceed_prune_limit() {
+return _lru_cache_type == LRUCacheType::SIZE ? mem_consumption() >
CACHE_MIN_FREE_SIZE
+ : get_usage() >
CACHE_MIN_FREE_NUMBER;
+}
// Try to prune the cache if expired.
void prune_stale() override {
@@ -142,8 +145,8 @@ public:
curtime);
};
-LOG(INFO) << fmt::format("[MemoryGC] {} prune stale start,
consumption {}",
- type_string(_type), mem_consumption());
+LOG(INFO) << fmt::format("[MemoryGC] {} prune stale start,
consumption {}, usage {}",
+ type_string(_type), mem_consumption(),
get_usage());
// Prune cache in lazy mode to save cpu and minimize the time
holding write lock
PrunedInfo pruned_info = _cache->prune_if(pred, true);
COUNTER_SET(_freed_entrys_counter, pruned_info.pruned_count);
@@ -154,10 +157,19 @@ public:
type_string(_type), _freed_entrys_counter->value(),
_freed_memory_counter->value(),
_prune_stale_number_counter->value());
} else {
-LOG(INFO) << fmt::format(
-"[MemoryGC] {} not need prune stale, consumption {} less
than "
-"CACHE_MIN_FREE_SIZE {}",
-type_string(_type), mem_consumption(),
CACHE_MIN_FREE_SIZE);
+if (_lru_cache_type == LRUCacheType::SIZE) {
+LOG(INFO) << fmt::format(
+"[MemoryGC] {} not need prune stale,
LRUCacheType::SIZE consumption {} "
+"less "
+"than CACHE_MIN_FREE_SIZE {}",
+type_string(_type), mem_consumption(),
CACHE_MIN_FREE_SIZE);
+} else if (_lru_cache_type == LRUCacheType::NUMBER) {
+LOG(INFO) << fmt::format(
+"[MemoryGC] {} not need prune stale,
LRUCacheType::NUMBER usage {} less "
+"than "
+"CACHE_MIN_FREE_NUMBER {}",
+type_string(_type), get_usage(),
CACHE_MIN_FREE_NUMBER);
+}
}
}
@@ -170,8 +182,8 @@ public:
if ((force && mem_consumption() != 0) || exceed_prune_limit()) {
COUNTER_SET(_cost_timer, (int64_t)0);
SCOPED_TIMER(_cost_timer);
-LOG(INFO) << fmt::format("[MemoryGC] {} prune all start,
consumption {}",
- type_string(_type), mem_consumption());
+LOG(INFO) << fmt::format("[MemoryGC] {} prune all start,
consumption {}, usage {}",
+ type_string(_type), mem_consumption(),
get_usage());
PrunedInfo pruned_info = _cache->prune();
COUNTER_SET(_freed_entrys_counter, pruned_info.pruned_count);
COUNTER_SET(_freed_memory_counter, pruned_info.pruned_size);
@@ -181,10 +193,18 @@ public:
type_string(_type), _freed_entrys_counter->value(),
_freed_memory_counter->value(),
_prune_all_number_counter->value(), force);
} else {
-LOG(INFO) <<
Re: [PR] [branch-2.1](memory) Fix prune all LRU Cache based on number [doris]
yiguolei merged PR #34736: URL: https://github.com/apache/doris/pull/34736 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] Release Note 2.0.10 [doris]
xiaokang commented on issue #34380: URL: https://github.com/apache/doris/issues/34380#issuecomment-2106107721 中文版本 ## 1 行为变更 无 ## 2 新功能 无 ## 3 改进和优化 - 增加了`read_only`和`super_read_only`变量以保持和MySQL兼容 - 仅在IO_ERROR的错误才把数据目录加入broken list，防止fd超限等错误导致误加入 - 基于外表CTAS创建新表时，把varchar类型转成string类型 - 支持把Paimon的Row类型映射成Doris的Struct类型 - 在创建tablet选择数据盘时，允许存在少量的倾斜 - 对`set replica drop`命令记录editlog，以防止在follower节点执行命令后，其状态显示不正确 - schema change 内存自适应避免内存超限 - 倒排索引中 unicode 分词器可以配置不使用停用词 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
svn commit: r69109 - in /dev/doris/2.0.10: ./ apache-doris-2.0.10-src.tar.gz apache-doris-2.0.10-src.tar.gz.asc apache-doris-2.0.10-src.tar.gz.sha512
Author: kxiao Date: Sun May 12 03:14:43 2024 New Revision: 69109 Log: doris 2.0.10 release Added: dev/doris/2.0.10/ dev/doris/2.0.10/apache-doris-2.0.10-src.tar.gz (with props) dev/doris/2.0.10/apache-doris-2.0.10-src.tar.gz.asc (with props) dev/doris/2.0.10/apache-doris-2.0.10-src.tar.gz.sha512 Added: dev/doris/2.0.10/apache-doris-2.0.10-src.tar.gz == Binary file - no diff available. Propchange: dev/doris/2.0.10/apache-doris-2.0.10-src.tar.gz -- svn:mime-type = application/x-gzip Added: dev/doris/2.0.10/apache-doris-2.0.10-src.tar.gz.asc == Binary file - no diff available. Propchange: dev/doris/2.0.10/apache-doris-2.0.10-src.tar.gz.asc -- svn:mime-type = application/pgp-signature Added: dev/doris/2.0.10/apache-doris-2.0.10-src.tar.gz.sha512 == --- dev/doris/2.0.10/apache-doris-2.0.10-src.tar.gz.sha512 (added) +++ dev/doris/2.0.10/apache-doris-2.0.10-src.tar.gz.sha512 Sun May 12 03:14:43 2024 @@ -0,0 +1 @@ +2caf55fbcc09972add4973fd27675f90c0f7d3129b40d397394abdb1ea407e511189ab4f5d8ca9474bd462581d14d7b4f9f806d4e0cba68a4c0761cea6a905a3 apache-doris-2.0.10-src.tar.gz - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [docs]Add workload system table [doris-website]
wangbo opened a new pull request, #639: URL: https://github.com/apache/doris-website/pull/639 Add workload system table -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
ByteYue commented on PR #34679: URL: https://github.com/apache/doris/pull/34679#issuecomment-2106092852 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [Fix](functions) fix strcmp return value #34565
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new ca9eb562330 [Fix](functions) fix strcmp return value #34565
ca9eb562330 is described below
commit ca9eb56233006846282a0aca1948d48ffc6b998e
Author: zclllyybb
AuthorDate: Sun May 12 09:27:32 2024 +0800
[Fix](functions) fix strcmp return value #34565
---
be/src/vec/common/string_ref.h | 24 +++--
.../string_functions/test_string_function.out | Bin 4162 -> 4217 bytes
.../string_functions/test_string_function.groovy | 3 +++
3 files changed, 16 insertions(+), 11 deletions(-)
diff --git a/be/src/vec/common/string_ref.h b/be/src/vec/common/string_ref.h
index 587924b45e3..72a7a911210 100644
--- a/be/src/vec/common/string_ref.h
+++ b/be/src/vec/common/string_ref.h
@@ -22,10 +22,10 @@
// IWYU pragma: no_include
#include
-#include
#include
#include
+#include
#include
#include
#include
@@ -38,7 +38,6 @@
#include "util/hash_util.hpp"
#include "util/slice.h"
#include "util/sse_util.hpp"
-#include "vec/common/string_ref.h"
#include "vec/common/unaligned.h"
#include "vec/core/types.h"
@@ -150,7 +149,8 @@ inline bool memequalSSE2Wide(const char* p1, const char*
p2, size_t size) {
// - s1/n1: ptr/len for the first string
// - s2/n2: ptr/len for the second string
// - len: min(n1, n2) - this can be more cheaply passed in by the caller
-inline int string_compare(const char* s1, int64_t n1, const char* s2, int64_t
n2, int64_t len) {
+PURE inline int string_compare(const char* s1, int64_t n1, const char* s2,
int64_t n2,
+ int64_t len) {
DCHECK_EQ(len, std::min(n1, n2));
#if defined(__SSE4_2__) || defined(__aarch64__)
while (len >= sse_util::CHARS_PER_128_BIT_REGISTER) {
@@ -199,8 +199,8 @@ struct StringRef {
std::string to_string() const { return std::string(data, size); }
std::string debug_string() const { return to_string(); }
-std::string_view to_string_view() const { return std::string_view(data,
size); }
-Slice to_slice() const { return doris::Slice(data, size); }
+std::string_view to_string_view() const { return {data, size}; }
+Slice to_slice() const { return {data, size}; }
// this is just for show, e.g. print data to error log, to avoid print
large string.
std::string to_prefix(size_t length) const { return std::string(data,
std::min(length, size)); }
@@ -209,7 +209,7 @@ struct StringRef {
operator std::string_view() const { return std::string_view {data, size}; }
StringRef substring(int start_pos, int new_len) const {
-return StringRef(data + start_pos, (new_len < 0) ? (size - start_pos)
: new_len);
+return {data + start_pos, (new_len < 0) ? (size - start_pos) :
new_len};
}
StringRef substring(int start_pos) const { return substring(start_pos,
size - start_pos); }
@@ -255,7 +255,9 @@ struct StringRef {
}
}
-return string_compare(this->data, this->size, other.data, other.size,
l);
+// string_compare doesn't have sign result
+int cmp_result = string_compare(this->data, this->size, other.data,
other.size, l);
+return (cmp_result > 0) - (cmp_result < 0);
}
void replace(const char* ptr, int len) {
@@ -363,8 +365,8 @@ inline size_t hash_less_than8(const char* data, size_t
size) {
inline size_t hash_less_than16(const char* data, size_t size) {
if (size > 8) {
-doris::vectorized::UInt64 a =
unaligned_load(data);
-doris::vectorized::UInt64 b =
unaligned_load(data + size - 8);
+auto a = unaligned_load(data);
+auto b = unaligned_load(data + size - 8);
return hash_len16(a, rotate_by_at_least1(b + size, size)) ^ b;
}
@@ -388,13 +390,13 @@ struct CRC32Hash {
size_t res = -1ULL;
do {
-doris::vectorized::UInt64 word =
unaligned_load(pos);
+auto word = unaligned_load(pos);
res = _mm_crc32_u64(res, word);
pos += 8;
} while (pos + 8 < end);
-doris::vectorized::UInt64 word =
unaligned_load(
+auto word = unaligned_load(
end - 8); /// I'm not sure if this is normal.
res = _mm_crc32_u64(res, word);
diff --git
a/regression-test/data/query_p0/sql_functions/string_functions/test_string_function.out
b/regression-test/data/query_p0/sql_functions/string_functions/test_string_function.out
index e3ca494c632..69ce99277fa 100644
Binary files
a/regression-test/data/query_p0/sql_functions/string_functions/test_string_function.out
and
b/regression-test/data/query_p0/sql_functions/string_functions/test_string_function.out
differ
diff --git
(doris) branch branch-2.1 updated: [fix](fs) Close local file writer when downloading via broker fs (#34714)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new a8be47f3ff9 [fix](fs) Close local file writer when downloading via
broker fs (#34714)
a8be47f3ff9 is described below
commit a8be47f3ff9140ba927e2bbb4029f2783e7feb92
Author: walter
AuthorDate: Sat May 11 19:55:12 2024 +0800
[fix](fs) Close local file writer when downloading via broker fs (#34714)
---
be/src/io/fs/broker_file_system.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/io/fs/broker_file_system.cpp
b/be/src/io/fs/broker_file_system.cpp
index 44582a0ff8d..6096a0540cb 100644
--- a/be/src/io/fs/broker_file_system.cpp
+++ b/be/src/io/fs/broker_file_system.cpp
@@ -416,7 +416,7 @@ Status BrokerFileSystem::download_impl(const Path&
remote_file, const Path& loca
RETURN_IF_ERROR(local_writer->append({read_buf.get(), read_len}));
} // file_handler should be closed before calculating checksum
-return Status::OK();
+return local_writer->close();
}
Status BrokerFileSystem::read_file(const TBrokerFD& fd, size_t offset, size_t
bytes_req,
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (20e2d2e2f87 -> cadbbdd2c08)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from 20e2d2e2f87 [Fix](executor)Fix workload thread start failed when follower convert to master add cadbbdd2c08 [fix](config) for compatibility issue of log dir config (#34734) No new revisions were added by this update. Summary of changes: be/src/common/config.cpp | 11 +-- .../src/main/java/org/apache/doris/common/ConfigBase.java | 8 +++- 2 files changed, 16 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
yiguolei merged PR #34734: URL: https://github.com/apache/doris/pull/34734 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](config) for compatibility issue of log dir config (#34734)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c6b469aad65 [fix](config) for compatibility issue of log dir config
(#34734)
c6b469aad65 is described below
commit c6b469aad657a50a6b8d0f11179dcc202485e619
Author: Mingyu Chen
AuthorDate: Sun May 12 09:44:15 2024 +0800
[fix](config) for compatibility issue of log dir config (#34734)
* [fix](config) for compatibility issue of log dir config
* 1
---
be/src/common/config.cpp | 11 +--
.../src/main/java/org/apache/doris/common/ConfigBase.java | 8 +++-
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index a773e1a8bfe..8055e3fe456 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1732,11 +1732,18 @@ std::vector> get_config_info()
{
std::vector _config;
_config.push_back(it.first);
+std::string config_val = it.second;
+// For compatibility, this PR #32933 change the log dir's config logic,
+// and deprecate the `sys_log_dir` config.
+if (it.first == "sys_log_dir" && config_val == "") {
+config_val = fmt::format("{}/log", std::getenv("DORIS_HOME"));
+}
+
_config.emplace_back(field_it->second.type);
if (0 == strcmp(field_it->second.type, "bool")) {
-_config.emplace_back(it.second == "1" ? "true" : "false");
+_config.emplace_back(config_val == "1" ? "true" : "false");
} else {
-_config.push_back(it.second);
+_config.push_back(config_val);
}
_config.emplace_back(field_it->second.valmutable ? "true" : "false");
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/ConfigBase.java
b/fe/fe-common/src/main/java/org/apache/doris/common/ConfigBase.java
index dd66fdcb0f5..3306f314f7d 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/ConfigBase.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/ConfigBase.java
@@ -376,7 +376,13 @@ public class ConfigBase {
if (matcher == null || matcher.match(confKey)) {
List config = Lists.newArrayList();
config.add(confKey);
-config.add(getConfValue(f));
+String value = getConfValue(f);
+// For compatibility, this PR #32933 change the log dir's
config logic,
+// and deprecate the `sys_log_dir` config.
+if (confKey.equals("sys_log_dir") &&
Strings.isNullOrEmpty(value)) {
+value = System.getenv("DORIS_HOME") + "/log";
+}
+config.add(value);
config.add(f.getType().getSimpleName());
config.add(String.valueOf(confField.mutable()));
config.add(String.valueOf(confField.masterOnly()));
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](join) For left semi/anti join without mark join conjunct and without other conjucnts, stop probing after matching one row [doris]
mrhhsg commented on code in PR #34703: URL: https://github.com/apache/doris/pull/34703#discussion_r1597532952 ## be/src/vec/exec/join/process_hash_table_probe_impl.h: ## @@ -258,7 +259,7 @@ Status ProcessHashTableProbe::do_process(HashTableType& hash need_null_map_for_probe && ignore_null > (hash_table_ctx.keys, hash_table_ctx.bucket_nums.data(), probe_index, build_index, probe_rows, _probe_indexs.data(), - _probe_visited, _build_indexs.data()); + _probe_visited, _build_indexs.data(), has_mark_join_conjunct); Review Comment: There are some cases already covered this code in `regression-test/suites/nereids_p0/join/test_mark_join.groovy`, `regression-test/suites/nereids_p0/subquery/subquery_unnesting.groovy`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.1](memory) Fix prune all LRU Cache based on number [doris]
doris-robot commented on PR #34736: URL: https://github.com/apache/doris/pull/34736#issuecomment-2106083292 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [branch-2.1](memory) Fix prune all LRU Cache based on number [doris]
xinyiZzz opened a new pull request, #34736: URL: https://github.com/apache/doris/pull/34736 ## Proposed changes pick #34601 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/01: [Fix](executor)Fix workload thread start failed when follower convert to master
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 20e2d2e2f87fbcc8c459605fb578215ed76dfb80
Author: wangbo
AuthorDate: Sat May 11 23:31:59 2024 +0800
[Fix](executor)Fix workload thread start failed when follower convert to
master
---
.../main/java/org/apache/doris/catalog/Env.java| 2 +-
.../resource/workloadgroup/WorkloadGroupMgr.java | 27 +++
.../WorkloadRuntimeStatusMgr.java | 64 ++--
.../WorkloadSchedPolicyMgr.java| 88 ++
.../workloadgroup/WorkloadGroupMgrTest.java| 33
.../data/workload_manager_p0/test_curd_wlg.out | 9 +++
regression-test/pipeline/p0/conf/fe.conf | 1 +
.../workload_manager_p0/test_curd_wlg.groovy | 6 ++
8 files changed, 89 insertions(+), 141 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index 0aa0da8ef36..4839769e0f8 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -1711,7 +1711,7 @@ public class Env {
dnsCache.start();
-workloadGroupMgr.startUpdateThread();
+workloadGroupMgr.start();
workloadSchedPolicyMgr.start();
workloadRuntimeStatusMgr.start();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
index a06413a94d4..2cfc59dfd2f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
@@ -34,6 +34,7 @@ import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.common.proc.BaseProcResult;
import org.apache.doris.common.proc.ProcResult;
+import org.apache.doris.common.util.MasterDaemon;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.persist.DropWorkloadGroupOperationLog;
import org.apache.doris.persist.gson.GsonPostProcessable;
@@ -64,7 +65,7 @@ import java.util.Map;
import java.util.Set;
import java.util.concurrent.locks.ReentrantReadWriteLock;
-public class WorkloadGroupMgr implements Writable, GsonPostProcessable {
+public class WorkloadGroupMgr extends MasterDaemon implements Writable,
GsonPostProcessable {
public static final String DEFAULT_GROUP_NAME = "normal";
@@ -90,22 +91,13 @@ public class WorkloadGroupMgr implements Writable,
GsonPostProcessable {
private final ResourceProcNode procNode = new ResourceProcNode();
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
-private Thread updatePropThread;
-
-public void startUpdateThread() {
-WorkloadGroupMgr wgMgr = this;
-updatePropThread = new Thread(() -> {
-Thread.currentThread().setName("reset-query-queue-prop");
-while (true) {
-try {
-wgMgr.resetQueryQueueProp();
-Thread.sleep(Config.query_queue_update_interval_ms);
-} catch (Throwable e) {
-LOG.warn("reset query queue failed ", e);
-}
-}
-});
-updatePropThread.start();
+@Override
+protected void runAfterCatalogReady() {
+try {
+resetQueryQueueProp();
+} catch (Throwable e) {
+LOG.warn("reset query queue failed ", e);
+}
}
public void resetQueryQueueProp() {
@@ -142,6 +134,7 @@ public class WorkloadGroupMgr implements Writable,
GsonPostProcessable {
}
public WorkloadGroupMgr() {
+super("workload-group-thread", Config.query_queue_update_interval_ms);
// if no fe image exist, we should append internal group here.
appendInternalWorkloadGroup();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
index ff2641d5f3c..de4810b65ac 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
@@ -19,7 +19,7 @@ package org.apache.doris.resource.workloadschedpolicy;
import org.apache.doris.catalog.Env;
import org.apache.doris.common.Config;
-import org.apache.doris.common.util.Daemon;
+import org.apache.doris.common.util.MasterDaemon;
import org.apache.doris.plugin.audit.AuditEvent;
import org.apache.doris.thrift.TQueryStatistics;
import
(doris) branch branch-2.1 updated (ad0e64bec17 -> 20e2d2e2f87)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git omit ad0e64bec17 [Fix](executor)Fix workload thread start failed when follower convert to master new 20e2d2e2f87 [Fix](executor)Fix workload thread start failed when follower convert to master This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (ad0e64bec17) \ N -- N -- N refs/heads/branch-2.1 (20e2d2e2f87) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java| 2 -- 1 file changed, 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Fix](functions) fix strcmp return value #34565
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5cc7dd5f52c [Fix](functions) fix strcmp return value #34565
5cc7dd5f52c is described below
commit 5cc7dd5f52c873985262b3ca5c8aef8f7749e751
Author: zclllyybb
AuthorDate: Sun May 12 09:27:32 2024 +0800
[Fix](functions) fix strcmp return value #34565
---
be/src/vec/common/string_ref.h | 24 +++--
.../string_functions/test_string_function.out | Bin 4162 -> 4217 bytes
.../string_functions/test_string_function.groovy | 3 +++
3 files changed, 16 insertions(+), 11 deletions(-)
diff --git a/be/src/vec/common/string_ref.h b/be/src/vec/common/string_ref.h
index 587924b45e3..72a7a911210 100644
--- a/be/src/vec/common/string_ref.h
+++ b/be/src/vec/common/string_ref.h
@@ -22,10 +22,10 @@
// IWYU pragma: no_include
#include
-#include
#include
#include
+#include
#include
#include
#include
@@ -38,7 +38,6 @@
#include "util/hash_util.hpp"
#include "util/slice.h"
#include "util/sse_util.hpp"
-#include "vec/common/string_ref.h"
#include "vec/common/unaligned.h"
#include "vec/core/types.h"
@@ -150,7 +149,8 @@ inline bool memequalSSE2Wide(const char* p1, const char*
p2, size_t size) {
// - s1/n1: ptr/len for the first string
// - s2/n2: ptr/len for the second string
// - len: min(n1, n2) - this can be more cheaply passed in by the caller
-inline int string_compare(const char* s1, int64_t n1, const char* s2, int64_t
n2, int64_t len) {
+PURE inline int string_compare(const char* s1, int64_t n1, const char* s2,
int64_t n2,
+ int64_t len) {
DCHECK_EQ(len, std::min(n1, n2));
#if defined(__SSE4_2__) || defined(__aarch64__)
while (len >= sse_util::CHARS_PER_128_BIT_REGISTER) {
@@ -199,8 +199,8 @@ struct StringRef {
std::string to_string() const { return std::string(data, size); }
std::string debug_string() const { return to_string(); }
-std::string_view to_string_view() const { return std::string_view(data,
size); }
-Slice to_slice() const { return doris::Slice(data, size); }
+std::string_view to_string_view() const { return {data, size}; }
+Slice to_slice() const { return {data, size}; }
// this is just for show, e.g. print data to error log, to avoid print
large string.
std::string to_prefix(size_t length) const { return std::string(data,
std::min(length, size)); }
@@ -209,7 +209,7 @@ struct StringRef {
operator std::string_view() const { return std::string_view {data, size}; }
StringRef substring(int start_pos, int new_len) const {
-return StringRef(data + start_pos, (new_len < 0) ? (size - start_pos)
: new_len);
+return {data + start_pos, (new_len < 0) ? (size - start_pos) :
new_len};
}
StringRef substring(int start_pos) const { return substring(start_pos,
size - start_pos); }
@@ -255,7 +255,9 @@ struct StringRef {
}
}
-return string_compare(this->data, this->size, other.data, other.size,
l);
+// string_compare doesn't have sign result
+int cmp_result = string_compare(this->data, this->size, other.data,
other.size, l);
+return (cmp_result > 0) - (cmp_result < 0);
}
void replace(const char* ptr, int len) {
@@ -363,8 +365,8 @@ inline size_t hash_less_than8(const char* data, size_t
size) {
inline size_t hash_less_than16(const char* data, size_t size) {
if (size > 8) {
-doris::vectorized::UInt64 a =
unaligned_load(data);
-doris::vectorized::UInt64 b =
unaligned_load(data + size - 8);
+auto a = unaligned_load(data);
+auto b = unaligned_load(data + size - 8);
return hash_len16(a, rotate_by_at_least1(b + size, size)) ^ b;
}
@@ -388,13 +390,13 @@ struct CRC32Hash {
size_t res = -1ULL;
do {
-doris::vectorized::UInt64 word =
unaligned_load(pos);
+auto word = unaligned_load(pos);
res = _mm_crc32_u64(res, word);
pos += 8;
} while (pos + 8 < end);
-doris::vectorized::UInt64 word =
unaligned_load(
+auto word = unaligned_load(
end - 8); /// I'm not sure if this is normal.
res = _mm_crc32_u64(res, word);
diff --git
a/regression-test/data/query_p0/sql_functions/string_functions/test_string_function.out
b/regression-test/data/query_p0/sql_functions/string_functions/test_string_function.out
index e3ca494c632..69ce99277fa 100644
Binary files
a/regression-test/data/query_p0/sql_functions/string_functions/test_string_function.out
and
b/regression-test/data/query_p0/sql_functions/string_functions/test_string_function.out
differ
diff --git
Re: [PR] [Fix](functions) fix strcmp return value [doris]
yiguolei merged PR #34565: URL: https://github.com/apache/doris/pull/34565 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
github-actions[bot] commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2106081495 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](debug) add more log in select_with_tablets to investigate [doris]
dataroaring commented on PR #34553: URL: https://github.com/apache/doris/pull/34553#issuecomment-2106080078 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bugfix] Fix mv column type is not changed when do schema change [doris]
dataroaring commented on PR #34598: URL: https://github.com/apache/doris/pull/34598#issuecomment-2106079847 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](regression-test) modify a key type tests [doris]
github-actions[bot] commented on PR #34600: URL: https://github.com/apache/doris/pull/34600#issuecomment-2106079756 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](regression-test) modify a key type tests [doris]
github-actions[bot] commented on PR #34600: URL: https://github.com/apache/doris/pull/34600#issuecomment-2106079747 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](storage-vault) Implement privilege management for storage vault [doris]
github-actions[bot] commented on PR #34638: URL: https://github.com/apache/doris/pull/34638#issuecomment-2106079452 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression-test](framework) support trino connector tests in multi be and fe env [doris]
shuke987 commented on PR #34731: URL: https://github.com/apache/doris/pull/34731#issuecomment-2106067910 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Tools) Fix the escape character '\' error in sql generated by tpcds tools scripts [doris]
github-actions[bot] commented on PR #25203: URL: https://github.com/apache/doris/pull/25203#issuecomment-2106065422 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](merge on write) Add primary keys consistency check for partial update [doris]
github-actions[bot] closed pull request #23443: [Enhancement](merge on write) Add primary keys consistency check for partial update URL: https://github.com/apache/doris/pull/23443 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [blog](sql-convertor) Update SQL Convertor blog to Newsletter (#625)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new d463c09cfa [blog](sql-convertor) Update SQL Convertor blog to Newsletter (#625) d463c09cfa is described below commit d463c09cfa4be392326d231f2ec3670cda1da14c Author: KassieZ <139741991+kass...@users.noreply.github.com> AuthorDate: Sun May 12 06:06:38 2024 +0700 [blog](sql-convertor) Update SQL Convertor blog to Newsletter (#625) --- ...pache-doris-sql-convertor-for-easy-migration.md | 61 +- src/components/recent-blogs/recent-blogs.data.ts | 9 ++-- src/constant/newsletter.data.ts| 15 +++--- 3 files changed, 48 insertions(+), 37 deletions(-) diff --git a/blog/from-presto-trino-clickhouse-and-hive-to-apache-doris-sql-convertor-for-easy-migration.md b/blog/from-presto-trino-clickhouse-and-hive-to-apache-doris-sql-convertor-for-easy-migration.md index cc9159809d..5e521adbc3 100644 --- a/blog/from-presto-trino-clickhouse-and-hive-to-apache-doris-sql-convertor-for-easy-migration.md +++ b/blog/from-presto-trino-clickhouse-and-hive-to-apache-doris-sql-convertor-for-easy-migration.md @@ -51,11 +51,11 @@ Currently, Presto, Trino, Hive, ClickHouse, and PostgreSQL dialects are supporte 2. On any frontend (FE) node, start the service using the following command. - - The SQL conversion service is stateless and can be started or stopped at any time. +- The SQL conversion service is stateless and can be started or stopped at any time. - - `port=5001` in the command specifies the service port. (You can use any available port.) +- `port=5001` in the command specifies the service port. (You can use any available port.) - - It is advisable to start a service individually for each FE node. +- It is advisable to start a service individually for each FE node. ```Shell nohup ./doris-sql-convertor-1.0.1-bin-x86 run --host=0.0.0.0 --port=5001 & @@ -73,7 +73,7 @@ After deployment, you can execute SQL directly in the command line. You can star - Presto -```SQL +```sql mysql> set sql_dialect=presto; Query OK, 0 rows affected (0.00 sec) @@ -100,7 +100,7 @@ mysql> SELECT cast(start_time as varchar(20)) as col1, - ClickHouse -```SQL +```sql mysql> set sql_dialect=clickhouse; Query OK, 0 rows affected (0.00 sec) @@ -132,35 +132,44 @@ For large-scale conversion, it is recommended to use the visual interface, on wh Follow these steps to deploy the visual conversion interface: 1. Environment: Docker, Docker-Compose + 2. Get Doris-SQL-Convertor Docker image + 3. Create a network for the image - 1. ```Bash - docker network create app_network - ``` + +```Bash +docker network create app_network +``` + 4. Decompress the package - 1. ```Bash - tar xzvf doris-sql-convertor-1.0.1.tar.gz - - cd doris-sql-convertor - ``` + +```Bash +tar xzvf doris-sql-convertor-1.0.1.tar.gz + +cd doris-sql-convertor +``` + 5. Edit the environment variables - 1. ```Bash - FLASK_APP=server/app.py - FLASK_DEBUG=1 - API_HOST=http://doris-sql-convertor-api:5000 - - # DOCKER TAG - API_TAG=latest - WEB_TAG=latest - ``` + +```Bash +FLASK_APP=server/app.py +FLASK_DEBUG=1 +API_HOST=http://doris-sql-convertor-api:5000 + +# DOCKER TAG +API_TAG=latest +WEB_TAG=latest +``` + 6. Start it up - 1. ```Bash - sh start.sh - ``` + +```Bash +sh start.sh +``` After deployment, you can access the service by `ip:8080` via your local browser. `8080` is the default port. You can change the mapping port. On the visual interface, you can select the source dialect type and target dialect type, and then click "Convert". -:::note +:::info Note 1. For batch conversion, each SQL statement should end with `; `. 2. The Doris SQL Convertor supports 239 UNION ALL conversions at most. diff --git a/src/components/recent-blogs/recent-blogs.data.ts b/src/components/recent-blogs/recent-blogs.data.ts index 4d6874ab13..3a104eb81e 100644 --- a/src/components/recent-blogs/recent-blogs.data.ts +++ b/src/components/recent-blogs/recent-blogs.data.ts @@ -1,4 +1,8 @@ export const
Error while running notifications feature from .asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [blog](sql-convertor) Update SQL Convertor blog to Newsletter [doris-website]
luzhijing merged PR #625: URL: https://github.com/apache/doris-website/pull/625 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running notifications feature from .asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [doc](ecosystem) replace some invalid links (#626)
This is an automated email from the ASF dual-hosted git repository.
luzhijing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 3b7e17806b [doc](ecosystem) replace some invalid links (#626)
3b7e17806b is described below
commit 3b7e17806b03d933a2b7095d380497f72f46f0c1
Author: Wanghuan
AuthorDate: Sun May 12 07:05:54 2024 +0800
[doc](ecosystem) replace some invalid links (#626)
-
Co-authored-by: wanghuan
---
docs/ecosystem/beats.md| 4 ++--
docs/ecosystem/datax.md| 4 ++--
docs/ecosystem/doris-kafka-connector.md| 2 +-
docs/ecosystem/flink-doris-connector.md| 6 +++---
docs/ecosystem/hive-bitmap-udf.md | 4 ++--
docs/ecosystem/hive-hll-udf.md | 6 +++---
docs/ecosystem/logstash.md | 4 ++--
docs/ecosystem/seatunnel.md| 10 +-
docs/ecosystem/spark-doris-connector.md| 4 ++--
.../docusaurus-plugin-content-docs/current/ecosystem/beats.md | 2 +-
.../docusaurus-plugin-content-docs/current/ecosystem/datax.md | 4 ++--
.../current/ecosystem/doris-kafka-connector.md | 2 +-
.../current/ecosystem/flink-doris-connector.md | 8
.../current/ecosystem/hive-bitmap-udf.md | 4 ++--
.../current/ecosystem/hive-hll-udf.md | 8
.../current/ecosystem/logstash.md | 4 ++--
.../current/ecosystem/seatunnel.md | 8
.../current/ecosystem/spark-doris-connector.md | 4 ++--
.../version-2.1/ecosystem/beats.md | 2 +-
.../version-2.1/ecosystem/datax.md | 4 ++--
.../version-2.1/ecosystem/doris-kafka-connector.md | 2 +-
.../version-2.1/ecosystem/flink-doris-connector.md | 6 +++---
.../version-2.1/ecosystem/hive-bitmap-udf.md | 4 ++--
.../version-2.1/ecosystem/hive-hll-udf.md | 6 +++---
.../version-2.1/ecosystem/logstash.md | 4 ++--
.../version-2.1/ecosystem/seatunnel.md | 8
.../version-2.1/ecosystem/spark-doris-connector.md | 4 ++--
versioned_docs/version-2.1/ecosystem/beats.md | 4 ++--
versioned_docs/version-2.1/ecosystem/datax.md | 4 ++--
versioned_docs/version-2.1/ecosystem/doris-kafka-connector.md | 2 +-
versioned_docs/version-2.1/ecosystem/flink-doris-connector.md | 6 +++---
versioned_docs/version-2.1/ecosystem/hive-bitmap-udf.md| 4 ++--
versioned_docs/version-2.1/ecosystem/hive-hll-udf.md | 6 +++---
versioned_docs/version-2.1/ecosystem/logstash.md | 4 ++--
versioned_docs/version-2.1/ecosystem/seatunnel.md | 10 +-
versioned_docs/version-2.1/ecosystem/spark-doris-connector.md | 4 ++--
36 files changed, 86 insertions(+), 86 deletions(-)
diff --git a/docs/ecosystem/beats.md b/docs/ecosystem/beats.md
index 83beb2b37b..db2c515c3c 100644
--- a/docs/ecosystem/beats.md
+++ b/docs/ecosystem/beats.md
@@ -30,7 +30,7 @@ This is an output implementation of [elastic
beats](https://github.com/elastic/b
This module is used to output data to Doris for elastic beats, use the HTTP
protocol to interact with the Doris FE Http interface, and import data through
Doris's stream load.
-[Learn more about Doris Stream Load
](../data-operate/import/import-way/stream-load-manual.md)
+[Learn more about Doris Stream Load
](../data-operate/import/stream-load-manual.md)
[Learn more about Doris](/)
@@ -104,7 +104,7 @@ Connection doris configuration:
| table | Table name
| |
| label_prefix | Import the identification prefix, the final generated ID is
*{label\_prefix}\_{db}\_{table}\_{time_stamp}*
| doris_beats |
| line_delimiter | Used to specify the newline character in the imported data,
the default is \n. Combinations of multiple characters can be used as newlines.
| \n |
-| headers| Users can pass in [stream-load import
parameters](../data-operate/import/import-way/stream-load-manual.md) through
the headers. | |
+| headers| Users can pass in [stream-load import
parameters](../data-operate/import/stream-load-manual.md) through the headers.
| |
Beats configuration:
diff --git a/docs/ecosystem/datax.md
Re: [PR] [doc](ecosystem) replace some invalid links [doris-website]
luzhijing merged PR #626: URL: https://github.com/apache/doris-website/pull/626 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [docs](aggregate) Fix Data Model - Aggregate EN docs and dead link of load manual (#627)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new de19026ac1 [docs](aggregate) Fix Data Model - Aggregate EN docs and dead link of load manual (#627) de19026ac1 is described below commit de19026ac15ed6e876e50086a289393cb04347fd Author: KassieZ <139741991+kass...@users.noreply.github.com> AuthorDate: Sun May 12 06:05:27 2024 +0700 [docs](aggregate) Fix Data Model - Aggregate EN docs and dead link of load manual (#627) --- .../current/data-operate/import/load-manual.md | 2 +- .../table-design/data-model/aggregate.md | 6 +- .../table-design/data-model/aggregate.md | 22 ++ 3 files changed, 20 insertions(+), 10 deletions(-) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/load-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/load-manual.md index a4d8da6404..ea9623d747 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/load-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/load-manual.md @@ -35,7 +35,7 @@ Doris 提供多种数据导入方案，可以针对不同的数据源进行选 | 数据源 | 导入方式 | | | | | 对象存储（s3）,HDFS | [使用 Broker 导入数据](../import/broker-load-manual) | -| 本地文件 | [Stream Load](../import/stream-load-manual), [MySQL Load](../import/mysql-load-manual) | +| 本地文件 | [Stream Load](../import/stream-load-manual), [MySQL Load](../../data-operate/import/mysql-load-manual) | | Kafka| [订阅 Kafka 数据](../import/routine-load-manual) | | Mysql、PostgreSQL，Oracle，SQLServer | [通过外部表同步数据](../import/insert-into-manual) | | 通过 JDBC 导入 | [使用 JDBC 同步数据](../../lakehouse/database/jdbc) | diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/table-design/data-model/aggregate.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/table-design/data-model/aggregate.md index c368baf200..ebf48ae590 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/table-design/data-model/aggregate.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/table-design/data-model/aggregate.md @@ -314,4 +314,8 @@ mysql> select sum_merge(k2) , group_concat_merge(k3)from aggstate where k1 != 2; 用户可以通过 agg_state 做出更细致的聚合函数操作。 -注意 agg_state 存在一定的性能开销。 \ No newline at end of file +:::caution 注意 + +agg_state 存在一定的性能开销。 + +::: \ No newline at end of file diff --git a/versioned_docs/version-2.1/table-design/data-model/aggregate.md b/versioned_docs/version-2.1/table-design/data-model/aggregate.md index 09d21aed78..8fa0cad826 100644 --- a/versioned_docs/version-2.1/table-design/data-model/aggregate.md +++ b/versioned_docs/version-2.1/table-design/data-model/aggregate.md @@ -34,19 +34,19 @@ Assume that the business has the following data table schema: | ColumnName | Type | AggregationType | Comment | | --- | | --- | --- | -| userid | LARGEINT | | user id | +| user_id | LARGEINT | | user id | | date| DATE | | date of data filling | -| City| VARCHAR (20) | | User City | +| city| VARCHAR (20) | | User City | | age | SMALLINT | | User age | | sex | TINYINT | | User gender | -| Last_visit_date | DATETIME | REPLACE | Last user access time | -| Cost| BIGINT | SUM | Total User Consumption | -| max dwell time | INT | MAX | Maximum user residence time | -| min dwell time | INT | MIN | User minimum residence time | +| last_visit_date | DATETIME | REPLACE | Last user access time | +| cost| BIGINT | SUM | Total User Consumption | +| max_dwell_time | INT | MAX | Maximum user residence time | +| min_dwell_time | INT | MIN | User minimum residence time | The corresponding to CREATE TABLE statement would be as follows (omitting the Partition and Distribution information): -``` +```sql CREATE DATABASE IF NOT EXISTS example_db; CREATE TABLE IF NOT EXISTS
Re: [PR] [docs](upgrade) Fix upgrade docs of version dev/2.1/2.0 [doris-website]
luzhijing merged PR #628: URL: https://github.com/apache/doris-website/pull/628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running notifications feature from .asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running notifications feature from .asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [docs](aggregate) Fix Data Model - Aggregate EN docs and dead link of load manual [doris-website]
luzhijing merged PR #627: URL: https://github.com/apache/doris-website/pull/627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [docs](upgrade) Fix upgrade docs of version dev/2.1/2.0 (#628)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 8712cb0eac [docs](upgrade) Fix upgrade docs of version dev/2.1/2.0 (#628) 8712cb0eac is described below commit 8712cb0eac70ee53df7267cbdb75582a3a69df6b Author: KassieZ <139741991+kass...@users.noreply.github.com> AuthorDate: Sun May 12 06:04:46 2024 +0700 [docs](upgrade) Fix upgrade docs of version dev/2.1/2.0 (#628) --- docs/admin-manual/cluster-management/upgrade.md| 3 +-- .../current/admin-manual/cluster-management/upgrade.md | 7 ++- .../version-2.0/admin-manual/cluster-management/upgrade.md | 7 ++- .../version-2.1/admin-manual/cluster-management/upgrade.md | 6 ++ .../version-2.0/admin-manual/cluster-management/upgrade.md | 4 ++-- .../version-2.1/admin-manual/cluster-management/upgrade.md | 3 +-- 6 files changed, 10 insertions(+), 20 deletions(-) diff --git a/docs/admin-manual/cluster-management/upgrade.md b/docs/admin-manual/cluster-management/upgrade.md index 584eaf3c9b..b1347d98da 100644 --- a/docs/admin-manual/cluster-management/upgrade.md +++ b/docs/admin-manual/cluster-management/upgrade.md @@ -31,11 +31,10 @@ To upgrade, please use the steps recommended in this chapter to upgrade the clus ## Doris Release Notes -:::tip When upgrading Doris, please follow the principle of **not skipping two minor versions** and upgrade sequentially. For example, if you are upgrading from version 0.15.x to 2.0.x, it is recommended to first upgrade to the latest version of 1.1, then upgrade to the latest version of 1.2, and finally upgrade to the latest version of 2.0. -::: + ## Upgrade Steps diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/cluster-management/upgrade.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/cluster-management/upgrade.md index 1687640714..d99b5c861a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/cluster-management/upgrade.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/cluster-management/upgrade.md @@ -31,12 +31,9 @@ under the License. ## Doris 版本说明 -:::tip - -Doris 升级请遵守**不要跨两个及以上关键节点版本升级**的原则，若要跨多个关键节点版本升级，先升级到最近的关键节点版本，随后再依次往后升级，若是非关键节点版本，则可忽略跳过。 +Doris 升级请遵守不要跨两个二位版本升级的原则，依次往后升级。 -关键节点版本：升级时必须要经历的版本，可能是单独一个版本，也可能是一个版本区间，如 `1.1.3 - 1.1.5`，则表示升级至该区间任意一版本即可继续后续升级。 -::: +比如从 0.15.x 升级到 2.0.x 版本，则建议先升级至 1.1 最新版本，然后升级到最新的 1.2 版本，最后升级到最新的 2.0 版本。 ## 升级步骤 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/admin-manual/cluster-management/upgrade.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/admin-manual/cluster-management/upgrade.md index 1687640714..d99b5c861a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/admin-manual/cluster-management/upgrade.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/admin-manual/cluster-management/upgrade.md @@ -31,12 +31,9 @@ under the License. ## Doris 版本说明 -:::tip - -Doris 升级请遵守**不要跨两个及以上关键节点版本升级**的原则，若要跨多个关键节点版本升级，先升级到最近的关键节点版本，随后再依次往后升级，若是非关键节点版本，则可忽略跳过。 +Doris 升级请遵守不要跨两个二位版本升级的原则，依次往后升级。 -关键节点版本：升级时必须要经历的版本，可能是单独一个版本，也可能是一个版本区间，如 `1.1.3 - 1.1.5`，则表示升级至该区间任意一版本即可继续后续升级。 -::: +比如从 0.15.x 升级到 2.0.x 版本，则建议先升级至 1.1 最新版本，然后升级到最新的 1.2 版本，最后升级到最新的 2.0 版本。 ## 升级步骤 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/cluster-management/upgrade.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/cluster-management/upgrade.md index 1687640714..be585ffc7d 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/cluster-management/upgrade.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/cluster-management/upgrade.md @@ -31,12 +31,10 @@ under the License. ## Doris 版本说明 -:::tip +Doris 升级请遵守不要跨两个二位版本升级的原则，依次往后升级。 -Doris 升级请遵守**不要跨两个及以上关键节点版本升级**的原则，若要跨多个关键节点版本升级，先升级到最近的关键节点版本，随后再依次往后升级，若是非关键节点版本，则可忽略跳过。 +比如从 0.15.x 升级到 2.0.x 版本，则建议先升级至 1.1 最新版本，然后升级到最新的 1.2 版本，最后升级到最新的 2.0 版本。 -关键节点版本：升级时必须要经历的版本，可能是单独一个版本，也可能是一个版本区间，如 `1.1.3 - 1.1.5`，则表示升级至该区间任意一版本即可继续后续升级。 -::: ## 升级步骤 diff --git a/versioned_docs/version-2.0/admin-manual/cluster-management/upgrade.md b/versioned_docs/version-2.0/admin-manual/cluster-management/upgrade.md index 584eaf3c9b..61be330b38 100644 --- a/versioned_docs/version-2.0/admin-manual/cluster-management/upgrade.md +++ b/versioned_docs/version-2.0/admin-manual/cluster-management/upgrade.md @@ -31,11 +31,11 @@ To upgrade, please use the steps recommended in this chapter to upgrade the clus ## Doris Release Notes -:::tip + When upgrading Doris, please follow the principle of **not skipping two minor versions** and upgrade sequentially. For example, if you are upgrading
Re: [PR] [Fix](functions) fix strcmp return value [doris]
github-actions[bot] commented on PR #34565: URL: https://github.com/apache/doris/pull/34565#issuecomment-2106046007 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](functions) fix strcmp return value [doris]
github-actions[bot] commented on PR #34565: URL: https://github.com/apache/doris/pull/34565#issuecomment-2106046012 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](be-ut) Fix annoying error message when using run-be-ut.sh (#34728)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2053380ebbf [fix](be-ut) Fix annoying error message when using
run-be-ut.sh (#34728)
2053380ebbf is described below
commit 2053380ebbfa2be866d0f6c229419c3e515c4e76
Author: zhiqiang
AuthorDate: Sun May 12 06:22:58 2024 +0800
[fix](be-ut) Fix annoying error message when using run-be-ut.sh (#34728)
---
run-be-ut.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/run-be-ut.sh b/run-be-ut.sh
index f9fcf9e9d53..453cd5d397a 100755
--- a/run-be-ut.sh
+++ b/run-be-ut.sh
@@ -276,7 +276,7 @@ rm -f "${UDF_RUNTIME_DIR}"/*
# clean all gcda file
while read -r gcda_file; do
rm "${gcda_file}"
-done < <(find "${DORIS_TEST_BINARY_DIR}" -name "*gcda")
+done < <(find "${CMAKE_BUILD_DIR}" -name "*gcda")
# prepare gtest output dir
GTEST_OUTPUT_DIR="${CMAKE_BUILD_DIR}/gtest_output"
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](be-ut) Fix annoying error message when using run-be-ut.sh [doris]
HappenLee merged PR #34728: URL: https://github.com/apache/doris/pull/34728 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](be-ut) Fix annoying error message when using run-be-ut.sh [doris]
github-actions[bot] commented on PR #34728: URL: https://github.com/apache/doris/pull/34728#issuecomment-2106044370 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](memory) Fix prune all LRU Cache based on number #34601
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dbece745f45 [fix](memory) Fix prune all LRU Cache based on number
#34601
dbece745f45 is described below
commit dbece745f45145b68ebf27cb1d0c3ec39bad125a
Author: Xinyi Zou
AuthorDate: Sun May 12 05:28:42 2024 +0800
[fix](memory) Fix prune all LRU Cache based on number #34601
---
be/src/runtime/memory/cache_policy.h | 1 +
be/src/runtime/memory/lru_cache_policy.h | 46 +++-
2 files changed, 34 insertions(+), 13 deletions(-)
diff --git a/be/src/runtime/memory/cache_policy.h
b/be/src/runtime/memory/cache_policy.h
index 7333fe4c937..75bf4881894 100644
--- a/be/src/runtime/memory/cache_policy.h
+++ b/be/src/runtime/memory/cache_policy.h
@@ -24,6 +24,7 @@
namespace doris {
static constexpr int32_t CACHE_MIN_FREE_SIZE = 67108864; // 64M
+static constexpr int32_t CACHE_MIN_FREE_NUMBER = 1024;
// Base of all caches. register to CacheManager when cache is constructed.
class CachePolicy {
diff --git a/be/src/runtime/memory/lru_cache_policy.h
b/be/src/runtime/memory/lru_cache_policy.h
index 8f0806e9da9..773817393c7 100644
--- a/be/src/runtime/memory/lru_cache_policy.h
+++ b/be/src/runtime/memory/lru_cache_policy.h
@@ -124,7 +124,10 @@ public:
uint64_t new_id() { return _cache->new_id(); };
// Subclass can override this method to determine whether to do the minor
or full gc
-virtual bool exceed_prune_limit() { return mem_consumption() >
CACHE_MIN_FREE_SIZE; }
+virtual bool exceed_prune_limit() {
+return _lru_cache_type == LRUCacheType::SIZE ? mem_consumption() >
CACHE_MIN_FREE_SIZE
+ : get_usage() >
CACHE_MIN_FREE_NUMBER;
+}
// Try to prune the cache if expired.
void prune_stale() override {
@@ -142,8 +145,8 @@ public:
curtime);
};
-LOG(INFO) << fmt::format("[MemoryGC] {} prune stale start,
consumption {}",
- type_string(_type), mem_consumption());
+LOG(INFO) << fmt::format("[MemoryGC] {} prune stale start,
consumption {}, usage {}",
+ type_string(_type), mem_consumption(),
get_usage());
// Prune cache in lazy mode to save cpu and minimize the time
holding write lock
PrunedInfo pruned_info = _cache->prune_if(pred, true);
COUNTER_SET(_freed_entrys_counter, pruned_info.pruned_count);
@@ -154,10 +157,19 @@ public:
type_string(_type), _freed_entrys_counter->value(),
_freed_memory_counter->value(),
_prune_stale_number_counter->value());
} else {
-LOG(INFO) << fmt::format(
-"[MemoryGC] {} not need prune stale, consumption {} less
than "
-"CACHE_MIN_FREE_SIZE {}",
-type_string(_type), mem_consumption(),
CACHE_MIN_FREE_SIZE);
+if (_lru_cache_type == LRUCacheType::SIZE) {
+LOG(INFO) << fmt::format(
+"[MemoryGC] {} not need prune stale,
LRUCacheType::SIZE consumption {} "
+"less "
+"than CACHE_MIN_FREE_SIZE {}",
+type_string(_type), mem_consumption(),
CACHE_MIN_FREE_SIZE);
+} else if (_lru_cache_type == LRUCacheType::NUMBER) {
+LOG(INFO) << fmt::format(
+"[MemoryGC] {} not need prune stale,
LRUCacheType::NUMBER usage {} less "
+"than "
+"CACHE_MIN_FREE_NUMBER {}",
+type_string(_type), get_usage(),
CACHE_MIN_FREE_NUMBER);
+}
}
}
@@ -170,8 +182,8 @@ public:
if ((force && mem_consumption() != 0) || exceed_prune_limit()) {
COUNTER_SET(_cost_timer, (int64_t)0);
SCOPED_TIMER(_cost_timer);
-LOG(INFO) << fmt::format("[MemoryGC] {} prune all start,
consumption {}",
- type_string(_type), mem_consumption());
+LOG(INFO) << fmt::format("[MemoryGC] {} prune all start,
consumption {}, usage {}",
+ type_string(_type), mem_consumption(),
get_usage());
PrunedInfo pruned_info = _cache->prune();
COUNTER_SET(_freed_entrys_counter, pruned_info.pruned_count);
COUNTER_SET(_freed_memory_counter, pruned_info.pruned_size);
@@ -181,10 +193,18 @@ public:
type_string(_type), _freed_entrys_counter->value(),
_freed_memory_counter->value(),
_prune_all_number_counter->value(), force);
} else {
-LOG(INFO) << fmt::format(
-
Re: [PR] [fix](memory) Fix prune all LRU Cache based on number [doris]
yiguolei merged PR #34601: URL: https://github.com/apache/doris/pull/34601 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](cloud) Use c++20 and unleash endian check [doris]
github-actions[bot] commented on code in PR #34735:
URL: https://github.com/apache/doris/pull/34735#discussion_r1597487093
##
cloud/src/meta-service/doris_txn.cpp:
##
@@ -29,7 +31,8 @@ int get_txn_id_from_fdb_ts(std::string_view fdb_vts, int64_t*
txn_id) {
// byte addr 0 1 2 3 4 5 6 7 8 9
int64_t ver = *reinterpret_cast(fdb_vts.data());
-// static_assert(std::endian::little); // Since c++20
+// TODO(gavin): implementation for big-endian or make it endian-independent
+static_assert(std::endian::native == std::endian::little); // Since c++20
Review Comment:
warning: no member named 'endian' in namespace 'std' [clang-diagnostic-error]
```cpp
static_assert(std::endian::native == std::endian::little); // Since c++20
^
```
##
cloud/src/meta-service/doris_txn.cpp:
##
@@ -29,7 +31,8 @@
// byte addr 0 1 2 3 4 5 6 7 8 9
int64_t ver = *reinterpret_cast(fdb_vts.data());
-// static_assert(std::endian::little); // Since c++20
+// TODO(gavin): implementation for big-endian or make it endian-independent
+static_assert(std::endian::native == std::endian::little); // Since c++20
Review Comment:
warning: no member named 'endian' in namespace 'std' [clang-diagnostic-error]
```cpp
static_assert(std::endian::native == std::endian::little); // Since c++20
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](cloud) Use c++20 and unleash endian check [doris]
gavinchou commented on PR #34735: URL: https://github.com/apache/doris/pull/34735#issuecomment-2105977101 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](cloud) Use c++20 and unleash endian check [doris]
doris-robot commented on PR #34735: URL: https://github.com/apache/doris/pull/34735#issuecomment-2105977061 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](storage-vault) Implement privilege management for storage vault [doris]
github-actions[bot] commented on PR #34638: URL: https://github.com/apache/doris/pull/34638#issuecomment-2105962766 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) Avoid unnecessary copies in the profile thrift. [doris]
doris-robot commented on PR #34720: URL: https://github.com/apache/doris/pull/34720#issuecomment-2105959482 TeamCity be ut coverage result: Function Coverage: 35.65% (8984/25202) Line Coverage: 27.31% (74254/271852) Region Coverage: 26.55% (38386/144558) Branch Coverage: 23.36% (19566/83752) Coverage Report: http://coverage.selectdb-in.cc/coverage/eab221738fe000fbd0279891a71a178cba8b4269_eab221738fe000fbd0279891a71a178cba8b4269/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) Avoid unnecessary copies in the profile thrift. [doris]
doris-robot commented on PR #34720: URL: https://github.com/apache/doris/pull/34720#issuecomment-2105959289 TPC-DS: Total hot run time: 187641 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit eab221738fe000fbd0279891a71a178cba8b4269, data reload: false query1 910 372 346 346 query2 6687252224642464 query3 6651209 210 209 query4 23942 21308 21243 21243 query5 4095415 427 415 query6 257 182 173 173 query7 4575286 290 286 query8 255 206 195 195 query9 8639239123542354 query10 453 257 262 257 query11 14654 14268 14152 14152 query12 138 87 87 87 query13 1643387 373 373 query14 10474 840877297729 query15 244 168 170 168 query16 8018263 265 263 query17 1726562 552 552 query18 2074279 267 267 query19 208 152 155 152 query20 94 85 83 83 query21 195 136 129 129 query22 5022479748664797 query23 34202 33531 33600 33531 query24 6875285528812855 query25 566 372 379 372 query26 697 158 151 151 query27 2001314 330 314 query28 3890210020742074 query29 855 608 595 595 query30 221 152 153 152 query31 965 777 776 776 query32 91 55 56 55 query33 476 249 243 243 query34 880 473 486 473 query35 774 685 685 685 query36 1058932 876 876 query37 102 63 66 63 query38 2959279027692769 query39 1635155315581553 query40 194 122 120 120 query41 41 40 38 38 query42 103 99 99 99 query43 583 566 568 566 query44 1074736 756 736 query45 270 236 243 236 query46 1048727 725 725 query47 1964189418891889 query48 373 311 296 296 query49 842 427 397 397 query50 763 391 386 386 query51 6925673567306730 query52 108 88 90 88 query53 351 279 277 277 query54 534 441 433 433 query55 73 71 71 71 query56 238 224 223 223 query57 1242113611711136 query58 216 190 198 190 query59 3526315833353158 query60 256 236 247 236 query61 97 88 87 87 query62 632 462 480 462 query63 312 278 276 276 query64 8418745173667366 query65 3132309530953095 query66 780 345 348 345 query67 15400 15138 15179 15138 query68 4559532 550 532 query69 470 302 305 302 query70 1199109811731098 query71 383 277 260 260 query72 7111258023682368 query73 695 323 319 319 query74 6577610960786078 query75 3349269126642664 query76 22421048961 961 query77 424 260 260 260 query78 10643 10194 10084 10084 query79 2502517 514 514 query80 1037429 429 429 query81 526 222 216 216 query82 747 91 92 91 query83 247 166 162 162 query84 254 90 86 86 query85 1268271 258 258 query86 527 303 325 303 query87 3310314231263126 query88 4256242424222422 query89 481 376 390 376 query90 2049186 183 183 query91 121 101 99 99 query92 64 47 48 47 query93 2111520 510 510 query94 1100180 191 180 query95 390 305 302 302 query96 591 271 269 269 query97 3174298830042988 query98 244 225 219 219 query99 1172905 914 905 Total cold run time: 272519 ms Total hot run time: 187641 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
doris-robot commented on PR #34679: URL: https://github.com/apache/doris/pull/34679#issuecomment-2105956751 TeamCity be ut coverage result: Function Coverage: 35.65% (8986/25205) Line Coverage: 27.32% (74259/271841) Region Coverage: 26.55% (38378/144541) Branch Coverage: 23.36% (19566/83748) Coverage Report: http://coverage.selectdb-in.cc/coverage/f1b67e4523c8c284d1eff456b611d9ec032b79e5_f1b67e4523c8c284d1eff456b611d9ec032b79e5/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) Avoid unnecessary copies in the profile thrift. [doris]
doris-robot commented on PR #34720: URL: https://github.com/apache/doris/pull/34720#issuecomment-2105956510 TPC-H: Total hot run time: 42000 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit eab221738fe000fbd0279891a71a178cba8b4269, data reload: false -- Round 1 -- q1 17604 437442804280 q2 2021198 189 189 q3 10451 123811591159 q4 10200 832 751 751 q5 7465275527572755 q6 230 136 136 136 q7 1033586 618 586 q8 9249214521072107 q9 9241679166946694 q10 9997389939213899 q11 454 252 244 244 q12 568 221 222 221 q13 17259 314532003145 q14 275 241 230 230 q15 514 472 484 472 q16 481 383 389 383 q17 997 725 745 725 q18 8293787378067806 q19 5114156415021502 q20 647 318 324 318 q21 5328420641024102 q22 365 303 296 296 Total cold run time: 117786 ms Total hot run time: 42000 ms - Round 2, with runtime_filter_mode=off - q1 4641437943944379 q2 375 268 271 268 q3 3171297929092909 q4 1940162016511620 q5 5509549454895489 q6 210 127 126 126 q7 2277198619671967 q8 3279344333793379 q9 8660867887328678 q10 3972377838003778 q11 594 491 479 479 q12 802 636 614 614 q13 16405 316031743160 q14 301 272 286 272 q15 520 470 489 470 q16 486 448 434 434 q17 1803147414731473 q18 7896761273577357 q19 1696160315561556 q20 1964176717481748 q21 8563481647304730 q22 584 506 503 503 Total cold run time: 75648 ms Total hot run time: 55389 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression-test](framework) support trino connector tests in multi be and fe env [doris]
shuke987 commented on PR #34731: URL: https://github.com/apache/doris/pull/34731#issuecomment-2105955985 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
doris-robot commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2105955618 TPC-DS: Total hot run time: 187802 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit c735b02d5ca717dfc6c7b99d5f71835fd0932336, data reload: false query1 903 370 358 358 query2 6846246022642264 query3 6636219 221 219 query4 25338 21171 21219 21171 query5 4093419 419 419 query6 259 186 200 186 query7 4589297 289 289 query8 236 197 188 188 query9 8495244024052405 query10 436 250 269 250 query11 14695 14299 14372 14299 query12 136 89 86 86 query13 1651381 371 371 query14 9715850683748374 query15 241 166 174 166 query16 8023267 269 267 query17 1721585 561 561 query18 2008286 274 274 query19 242 156 159 156 query20 92 89 84 84 query21 197 135 128 128 query22 5039482047654765 query23 33896 33500 33623 33500 query24 7369294129482941 query25 566 368 382 368 query26 701 157 155 155 query27 2116321 314 314 query28 5364207920442044 query29 852 594 604 594 query30 253 152 150 150 query31 964 745 729 729 query32 86 52 55 52 query33 529 267 247 247 query34 864 490 480 480 query35 770 691 704 691 query36 1078894 936 894 query37 107 65 65 65 query38 2904276027242724 query39 1590155415451545 query40 198 123 121 121 query41 42 38 37 37 query42 106 98 99 98 query43 564 554 538 538 query44 1083722 738 722 query45 270 246 251 246 query46 1071755 716 716 query47 1955187318981873 query48 374 296 299 296 query49 846 392 395 392 query50 773 395 382 382 query51 6886671867586718 query52 100 89 97 89 query53 356 301 305 301 query54 540 435 432 432 query55 73 71 70 70 query56 236 219 221 219 query57 1239117011321132 query58 213 193 201 193 query59 3356308032213080 query60 258 235 234 234 query61 100 86 86 86 query62 626 459 466 459 query63 308 289 290 289 query64 8451737074357370 query65 3138311630793079 query66 788 338 351 338 query67 15409 14881 14960 14881 query68 4516546 522 522 query69 475 310 301 301 query70 1195114411471144 query71 369 274 270 270 query72 7208256923922392 query73 713 332 329 329 query74 6507610461106104 query75 3329262126142614 query76 22521039945 945 query77 406 263 259 259 query78 10660 10121 10178 10121 query79 2324520 520 520 query80 1079425 437 425 query81 515 219 229 219 query82 734 96 93 93 query83 235 164 170 164 query84 252 84 87 84 query85 1323268 257 257 query86 528 324 285 285 query87 3346318130993099 query88 4177244524472445 query89 475 377 379 377 query90 2028188 193 188 query91 124 99 97 97 query92 65 51 49 49 query93 1846502 498 498 query94 1167180 187 180 query95 393 306 309 306 query96 597 270 268 268 query97 3147300130033001 query98 231 214 217 214 query99 1196910 868 868 Total cold run time: 274371 ms Total hot run time: 187802 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
(doris) branch master updated: [Fix](executor)Fix workload thread start failed when follower convert to master (#34695)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fad1a211dbf [Fix](executor)Fix workload thread start failed when
follower convert to master (#34695)
fad1a211dbf is described below
commit fad1a211dbf902befb42f8b6d2d41bd5e34cbce5
Author: wangbo
AuthorDate: Sun May 12 00:53:06 2024 +0800
[Fix](executor)Fix workload thread start failed when follower convert to
master (#34695)
* Fix workload thread start
* Fix fix test
---
.../main/java/org/apache/doris/catalog/Env.java| 2 +-
.../resource/workloadgroup/WorkloadGroupMgr.java | 27 +++
.../WorkloadRuntimeStatusMgr.java | 68 +++--
.../WorkloadSchedPolicyMgr.java| 88 ++
.../workloadgroup/WorkloadGroupMgrTest.java| 33
.../data/workload_manager_p0/test_curd_wlg.out | 9 +++
regression-test/pipeline/p0/conf/fe.conf | 1 +
.../workload_manager_p0/test_curd_wlg.groovy | 6 ++
8 files changed, 91 insertions(+), 143 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index 47bb945911d..05b65986753 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -1741,7 +1741,7 @@ public class Env {
dnsCache.start();
-workloadGroupMgr.startUpdateThread();
+workloadGroupMgr.start();
workloadSchedPolicyMgr.start();
workloadRuntimeStatusMgr.start();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
index 41a4949815e..24b0d378e5b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
@@ -34,6 +34,7 @@ import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.common.proc.BaseProcResult;
import org.apache.doris.common.proc.ProcResult;
+import org.apache.doris.common.util.MasterDaemon;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.persist.DropWorkloadGroupOperationLog;
import org.apache.doris.persist.gson.GsonPostProcessable;
@@ -64,7 +65,7 @@ import java.util.Map;
import java.util.Set;
import java.util.concurrent.locks.ReentrantReadWriteLock;
-public class WorkloadGroupMgr implements Writable, GsonPostProcessable {
+public class WorkloadGroupMgr extends MasterDaemon implements Writable,
GsonPostProcessable {
public static final String DEFAULT_GROUP_NAME = "normal";
@@ -90,22 +91,13 @@ public class WorkloadGroupMgr implements Writable,
GsonPostProcessable {
private final ResourceProcNode procNode = new ResourceProcNode();
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
-private Thread updatePropThread;
-
-public void startUpdateThread() {
-WorkloadGroupMgr wgMgr = this;
-updatePropThread = new Thread(() -> {
-Thread.currentThread().setName("reset-query-queue-prop");
-while (true) {
-try {
-wgMgr.resetQueryQueueProp();
-Thread.sleep(Config.query_queue_update_interval_ms);
-} catch (Throwable e) {
-LOG.warn("reset query queue failed ", e);
-}
-}
-});
-updatePropThread.start();
+@Override
+protected void runAfterCatalogReady() {
+try {
+resetQueryQueueProp();
+} catch (Throwable e) {
+LOG.warn("reset query queue failed ", e);
+}
}
public void resetQueryQueueProp() {
@@ -142,6 +134,7 @@ public class WorkloadGroupMgr implements Writable,
GsonPostProcessable {
}
public WorkloadGroupMgr() {
+super("workload-group-thread", Config.query_queue_update_interval_ms);
// if no fe image exist, we should append internal group here.
appendInternalWorkloadGroup();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
index 983643ec49b..f1b87196516 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
@@ -19,7 +19,7 @@ package org.apache.doris.resource.workloadschedpolicy;
import
Re: [PR] [Fix](executor)Fix workload thread start failed when follower convert to master [doris]
yiguolei merged PR #34695: URL: https://github.com/apache/doris/pull/34695 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
doris-robot commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2105952984 TPC-H: Total hot run time: 41837 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit c735b02d5ca717dfc6c7b99d5f71835fd0932336, data reload: false -- Round 1 -- q1 17603 438542124212 q2 2024192 199 192 q3 10424 131312321232 q4 10195 783 843 783 q5 7444273328302733 q6 227 133 137 133 q7 1028604 605 604 q8 9240213520842084 q9 10290 671067396710 q10 9074387838433843 q11 452 255 237 237 q12 417 246 229 229 q13 17254 324231623162 q14 276 224 231 224 q15 526 484 474 474 q16 528 420 406 406 q17 1019774 638 638 q18 8285773577657735 q19 4088154015321532 q20 652 331 321 321 q21 5240421540634063 q22 375 290 294 290 Total cold run time: 116661 ms Total hot run time: 41837 ms - Round 2, with runtime_filter_mode=off - q1 4519447944004400 q2 384 275 265 265 q3 3179290929332909 q4 1955171416191619 q5 5309546454745464 q6 215 128 131 128 q7 2333196819341934 q8 3245346733743374 q9 8572867587398675 q10 4099386137383738 q11 586 479 495 479 q12 772 596 618 596 q13 16083 313332043133 q14 314 292 249 249 q15 525 488 481 481 q16 483 430 457 430 q17 1812150915411509 q18 7939745273427342 q19 1647157415301530 q20 1978177817311731 q21 11422 485048504850 q22 603 500 494 494 Total cold run time: 77974 ms Total hot run time: 55330 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
doris-robot commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2105950751 TeamCity be ut coverage result: Function Coverage: 35.65% (8985/25201) Line Coverage: 27.32% (74272/271843) Region Coverage: 26.56% (38389/144556) Branch Coverage: 23.37% (19571/83756) Coverage Report: http://coverage.selectdb-in.cc/coverage/c735b02d5ca717dfc6c7b99d5f71835fd0932336_c735b02d5ca717dfc6c7b99d5f71835fd0932336/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) improve spill log printing [doris]
doris-robot commented on PR #34729: URL: https://github.com/apache/doris/pull/34729#issuecomment-2105948382 TPC-DS: Total hot run time: 187430 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 4250b140d95d3f381615b1080748866fa17d2dda, data reload: false query1 921 367 354 354 query2 6456244425232444 query3 6650212 212 212 query4 23159 21306 21273 21273 query5 4105422 413 413 query6 260 175 181 175 query7 4580300 299 299 query8 245 202 191 191 query9 8465239924032399 query10 438 253 274 253 query11 15127 14240 14089 14089 query12 134 90 90 90 query13 1641369 375 369 query14 9008771084607710 query15 246 171 178 171 query16 8104271 270 270 query17 1751583 551 551 query18 2106285 275 275 query19 204 157 154 154 query20 98 88 89 88 query21 198 125 128 125 query22 5092487948874879 query23 34212 33554 33576 33554 query24 8090291729732917 query25 582 425 379 379 query26 687 154 151 151 query27 2141310 330 310 query28 5565206420642064 query29 837 618 584 584 query30 229 150 158 150 query31 972 752 743 743 query32 89 51 52 51 query33 553 264 246 246 query34 880 477 470 470 query35 777 684 671 671 query36 1065936 939 936 query37 98 66 64 64 query38 2906278527512751 query39 1644169815461546 query40 195 123 122 122 query41 40 39 37 37 query42 102 98 96 96 query43 564 561 542 542 query44 1047717 716 716 query45 266 244 257 244 query46 1099711 721 711 query47 1974186619221866 query48 372 289 298 289 query49 856 404 400 400 query50 773 398 375 375 query51 6910676867866768 query52 102 98 87 87 query53 350 276 276 276 query54 589 423 429 423 query55 73 71 72 71 query56 248 213 225 213 query57 1219115311581153 query58 214 201 196 196 query59 3500316332823163 query60 258 228 231 228 query61 87 88 111 88 query62 622 467 490 467 query63 308 278 292 278 query64 8454732473517324 query65 3133309030513051 query66 795 346 345 345 query67 15369 15110 14895 14895 query68 4492532 529 529 query69 477 306 310 306 query70 1171107111541071 query71 357 271 260 260 query72 7526252223542354 query73 709 332 326 326 query74 6445611661516116 query75 3341264125882588 query76 2335997 1008997 query77 389 265 260 260 query78 10698 10328 10177 10177 query79 2259504 558 504 query80 1218421 429 421 query81 524 227 218 218 query82 682 95 88 88 query83 243 193 166 166 query84 238 87 86 86 query85 2036278 260 260 query86 494 318 324 318 query87 3308314131473141 query88 4264241824282418 query89 467 379 372 372 query90 2030188 183 183 query91 124 96 96 96 query92 63 48 49 48 query93 2581507 501 501 query94 1266184 192 184 query95 383 307 310 307 query96 592 268 266 266 query97 3180296829782968 query98 246 218 212 212 query99 1139960 894 894 Total cold run time: 275111 ms Total hot run time: 187430 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [improvement](spill) improve spill log printing [doris]
doris-robot commented on PR #34729: URL: https://github.com/apache/doris/pull/34729#issuecomment-2105945789 TPC-H: Total hot run time: 41897 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 4250b140d95d3f381615b1080748866fa17d2dda, data reload: false -- Round 1 -- q1 17609 485042574257 q2 2016193 184 184 q3 10492 128511701170 q4 10201 762 823 762 q5 7467273727002700 q6 234 131 135 131 q7 1039643 615 615 q8 9229217920972097 q9 9127670068116700 q10 9229389038463846 q11 446 253 250 250 q12 501 218 217 217 q13 17195 316332713163 q14 292 257 215 215 q15 518 469 481 469 q16 475 382 413 382 q17 1001727 748 727 q18 8386777077377737 q19 5766157615391539 q20 635 341 318 318 q21 5444413243224132 q22 368 286 288 286 Total cold run time: 117670 ms Total hot run time: 41897 ms - Round 2, with runtime_filter_mode=off - q1 4595441644274416 q2 372 274 268 268 q3 3137299828192819 q4 1914162016771620 q5 5504552655205520 q6 219 126 122 122 q7 2354195820131958 q8 3217337833853378 q9 8688872885298529 q10 3959387038803870 q11 580 472 495 472 q12 815 617 647 617 q13 16187 320931003100 q14 288 262 267 262 q15 518 481 465 465 q16 464 439 426 426 q17 1799147714601460 q18 7570746673457345 q19 1681149415081494 q20 1977178817731773 q21 8793478548874785 q22 597 485 516 485 Total cold run time: 75228 ms Total hot run time: 55184 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
github-actions[bot] commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2105944900 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](decimal) improve overflow error message [doris]
doris-robot commented on PR #34689: URL: https://github.com/apache/doris/pull/34689#issuecomment-2105943982 TPC-DS: Total hot run time: 187039 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit cd08a0cdf8f1b6da41cb0b73120098ccecac0a4a, data reload: false query1 766 365 352 352 query2 6421242523322332 query3 6407213 226 213 query4 23676 21455 21458 21455 query5 3941407 413 407 query6 244 173 175 173 query7 4402291 288 288 query8 245 197 200 197 query9 8229243524272427 query10 410 244 251 244 query11 14838 14162 14092 14092 query12 138 84 87 84 query13 1577372 371 371 query14 9936788677027702 query15 254 180 174 174 query16 8154266 275 266 query17 1840568 558 558 query18 2120281 279 279 query19 338 156 157 156 query20 93 84 84 84 query21 194 129 128 128 query22 5115491648974897 query23 34415 33572 4 4 query24 10608 286429162864 query25 583 360 353 353 query26 843 156 151 151 query27 2182317 317 317 query28 6061203920402039 query29 847 621 589 589 query30 233 148 149 148 query31 942 715 736 715 query32 89 49 55 49 query33 754 247 240 240 query34 1038481 483 481 query35 802 672 665 665 query36 1045907 860 860 query37 102 67 67 67 query38 2884277427322732 query39 1624155015501550 query40 196 123 123 123 query41 43 38 38 38 query42 106 93 93 93 query43 581 542 521 521 query44 1190717 750 717 query45 276 250 246 246 query46 1085748 718 718 query47 1963192319081908 query48 385 283 292 283 query49 851 392 381 381 query50 759 376 375 375 query51 6872672267306722 query52 97 84 89 84 query53 384 278 285 278 query54 899 426 438 426 query55 74 72 72 72 query56 235 217 222 217 query57 1249121012151210 query58 224 203 197 197 query59 3306315031123112 query60 257 235 235 235 query61 93 85 91 85 query62 652 492 468 468 query63 315 290 282 282 query64 8605740074007400 query65 3135309130443044 query66 759 328 342 328 query67 15316 15366 14860 14860 query68 6123540 535 535 query69 550 304 305 304 query70 1154111511091109 query71 479 267 268 267 query72 8403252123392339 query73 731 320 320 320 query74 6474615061896150 query75 3987263826302630 query76 3886987 1008987 query77 575 265 261 261 query78 10800 10168 10123 10123 query79 2155503 512 503 query80 774 436 424 424 query81 497 215 217 215 query82 129691 96 91 query83 182 164 166 164 query84 241 88 84 84 query85 1373263 262 262 query86 460 284 297 284 query87 3277311531253115 query88 3959240723932393 query89 467 375 377 375 query90 1969193 189 189 query91 121 98 141 98 query92 54 47 47 47 query93 2050506 498 498 query94 1207183 176 176 query95 396 306 297 297 query96 592 273 263 263 query97 3200298730272987 query98 244 208 217 208 query99 1163885 898 885 Total cold run time: 283136 ms Total hot run time: 187039 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
morningman commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2105943696 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
github-actions[bot] commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2105943047 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression-test](framework) support trino connector tests in multi be and fe env [doris]
shuke987 commented on PR #34731: URL: https://github.com/apache/doris/pull/34731#issuecomment-2105942279 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
github-actions[bot] commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2105941542 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](decimal) improve overflow error message [doris]
doris-robot commented on PR #34689: URL: https://github.com/apache/doris/pull/34689#issuecomment-2105941151 TPC-H: Total hot run time: 41489 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit cd08a0cdf8f1b6da41cb0b73120098ccecac0a4a, data reload: false -- Round 1 -- q1 17620 435942354235 q2 2017183 181 181 q3 10493 114311711143 q4 10192 757 817 757 q5 7475275427082708 q6 231 131 130 130 q7 933 523 534 523 q8 9249215221032103 q9 10028 729873537298 q10 9271389438373837 q11 451 233 246 233 q12 496 229 221 221 q13 17441 316332443163 q14 257 205 219 205 q15 515 460 460 460 q16 474 394 411 394 q17 973 644 714 644 q18 8336786877677767 q19 4296154515521545 q20 677 315 307 307 q21 5255421933623362 q22 355 295 273 273 Total cold run time: 117035 ms Total hot run time: 41489 ms - Round 2, with runtime_filter_mode=off - q1 4410425442574254 q2 358 256 259 256 q3 2974276727732767 q4 1856161316351613 q5 5325530253245302 q6 210 123 120 120 q7 1701135313871353 q8 3215337233703370 q9 9452937394629373 q10 4085374138703741 q11 598 483 505 483 q12 802 632 639 632 q13 16037 314631833146 q14 319 264 249 249 q15 538 475 484 475 q16 490 446 451 446 q17 1813155915261526 q18 8490850380698069 q19 1753158315411541 q20 2087190419061904 q21 8728507250005000 q22 616 533 532 532 Total cold run time: 75857 ms Total hot run time: 56152 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
github-actions[bot] commented on PR #34679: URL: https://github.com/apache/doris/pull/34679#issuecomment-2105941133 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
github-actions[bot] commented on code in PR #34679:
URL: https://github.com/apache/doris/pull/34679#discussion_r1597470210
##
be/src/io/fs/local_file_writer.cpp:
##
@@ -159,7 +159,8 @@ Status LocalFileWriter::appendv(const Slice* data, size_t
data_cnt) {
return Status::OK();
}
-Status LocalFileWriter::finalize() {
+// TODO(ByteYue): Refactor this function as FileWriter::flush()
+Status LocalFileWriter::_finalize() {
Review Comment:
warning: method '_finalize' can be made const
[readability-make-member-function-const]
```suggestion
Status LocalFileWriter::_finalize() const {
```
be/src/io/fs/local_file_writer.h:43:
```diff
- Status _finalize();
+ Status _finalize() const;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
github-actions[bot] commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2105939821 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
ByteYue commented on PR #34679: URL: https://github.com/apache/doris/pull/34679#issuecomment-2105939424 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](executor)Fix workload thread start failed when follower convert to master [doris]
github-actions[bot] commented on PR #34695: URL: https://github.com/apache/doris/pull/34695#issuecomment-2105939403 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) fuzzy spill and fix bugs [doris]
doris-robot commented on PR #33291: URL: https://github.com/apache/doris/pull/33291#issuecomment-2105939223 TPC-DS: Total hot run time: 185549 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit cd05004b3d8d43ab32e31036d6a66ad17b8655c8, data reload: false query1 946 355 337 337 query2 6467236722732273 query3 6687211 208 208 query4 22910 21282 21267 21267 query5 4132418 416 416 query6 265 193 197 193 query7 4582282 282 282 query8 232 181 183 181 query9 8458239823782378 query10 434 262 244 244 query11 14806 14318 14056 14056 query12 139 91 88 88 query13 1646380 375 375 query14 9700837767066706 query15 246 162 178 162 query16 8167272 259 259 query17 1843540 540 540 query18 2108280 267 267 query19 254 146 145 145 query20 89 83 82 82 query21 193 131 129 129 query22 5117493649484936 query23 34201 33665 33492 33492 query24 12013 289528402840 query25 643 365 361 361 query26 1747172 149 149 query27 2887313 308 308 query28 7341205020312031 query29 1002604 596 596 query30 298 148 156 148 query31 1006728 722 722 query32 94 52 54 52 query33 743 239 258 239 query34 1038460 469 460 query35 821 670 667 667 query36 1069913 941 913 query37 259 64 68 64 query38 2917279327492749 query39 1662153615841536 query40 275 122 122 122 query41 40 38 40 38 query42 102 93 97 93 query43 562 527 535 527 query44 1212711 732 711 query45 263 246 247 246 query46 1101714 720 714 query47 1959190918551855 query48 379 297 294 294 query49 1183385 388 385 query50 775 392 376 376 query51 6933680467406740 query52 102 86 92 86 query53 354 290 274 274 query54 986 432 431 431 query55 75 71 74 71 query56 248 216 222 216 query57 1217113711641137 query58 226 194 197 194 query59 3357339030133013 query60 260 235 239 235 query61 89 116 86 86 query62 679 466 458 458 query63 310 282 283 282 query64 9732730373367303 query65 3166307631083076 query66 1387330 337 330 query67 15372 15381 15029 15029 query68 4634536 540 536 query69 481 311 356 311 query70 1092112711291127 query71 415 264 274 264 query72 7214254223892389 query73 706 323 317 317 query74 6499621660216021 query75 3349259426142594 query76 2911991 1056991 query77 417 263 261 261 query78 10709 10128 10013 10013 query79 1956515 516 515 query80 1055437 423 423 query81 519 219 216 216 query82 693 93 90 90 query83 243 166 170 166 query84 234 86 85 85 query85 1975272 276 272 query86 496 277 321 277 query87 3289311931193119 query88 4186239224072392 query89 482 373 383 373 query90 1999186 181 181 query91 122 97 96 96 query92 61 46 49 46 query93 2217512 498 498 query94 1231182 188 182 query95 390 296 308 296 query96 591 270 273 270 query97 3155303629422942 query98 234 212 227 212 query99 1262901 917 901 Total cold run time: 286323 ms Total hot run time: 185549 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
morningman commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2105939245 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](config) for compatibility issue of log dir config [doris]
doris-robot commented on PR #34734: URL: https://github.com/apache/doris/pull/34734#issuecomment-2105938654 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](config) for compatibility issue of log dir config [doris]
morningman opened a new pull request, #34734: URL: https://github.com/apache/doris/pull/34734 ## Proposed changes PR #32933 change the config log dir logic, but result result empty `sys_log_dir` when calling `/rest/v2/manager/node/configuration_info` rest API. This PR make it compatible with old version, return the real log dir as `sys_log_dir`'s value in rest API ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) improve spill log printing [doris]
github-actions[bot] commented on PR #34729: URL: https://github.com/apache/doris/pull/34729#issuecomment-2105937683 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) fuzzy spill and fix bugs [doris]
doris-robot commented on PR #33291: URL: https://github.com/apache/doris/pull/33291#issuecomment-2105936737 TPC-H: Total hot run time: 40763 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit cd05004b3d8d43ab32e31036d6a66ad17b8655c8, data reload: false -- Round 1 -- q1 17658 532042484248 q2 2022195 194 194 q3 10658 116712331167 q4 10363 770 773 770 q5 7540271226532653 q6 218 133 138 133 q7 1021606 580 580 q8 9628212521222122 q9 9148665865676567 q10 9089370037153700 q11 460 241 240 240 q12 552 214 214 214 q13 17780 298029342934 q14 261 229 230 229 q15 512 472 467 467 q16 532 389 404 389 q17 955 709 695 695 q18 7939740373767376 q19 4372152115221521 q20 647 313 302 302 q21 5000398041073980 q22 351 284 282 282 Total cold run time: 116706 ms Total hot run time: 40763 ms - Round 2, with runtime_filter_mode=off - q1 4358428242824282 q2 371 278 271 271 q3 2982280327352735 q4 1879153715821537 q5 5240528952575257 q6 208 124 125 124 q7 2226187218361836 q8 3178330533263305 q9 8309830982928292 q10 3899374036643664 q11 575 487 485 485 q12 771 588 599 588 q13 17452 296029772960 q14 271 270 252 252 q15 510 462 474 462 q16 471 425 411 411 q17 1767148914371437 q18 7663751372767276 q19 1649156915701569 q20 1956175517651755 q21 5047491849094909 q22 567 505 512 505 Total cold run time: 71349 ms Total hot run time: 53912 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) improve spill log printing [doris]
jacktengg commented on PR #34729: URL: https://github.com/apache/doris/pull/34729#issuecomment-2105936391 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [env](thrift) Generate move constructors and assignment operators. [doris]
github-actions[bot] commented on PR #34693: URL: https://github.com/apache/doris/pull/34693#issuecomment-2105936211 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [env](thrift) Generate move constructors and assignment operators. [doris]
github-actions[bot] commented on PR #34693: URL: https://github.com/apache/doris/pull/34693#issuecomment-2105936200 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](join) For left semi/anti join without mark join conjunct and without other conjucnts, stop probing after matching one row [doris]
yiguolei commented on code in PR #34703: URL: https://github.com/apache/doris/pull/34703#discussion_r1597466795 ## be/src/vec/exec/join/process_hash_table_probe_impl.h: ## @@ -258,7 +259,7 @@ Status ProcessHashTableProbe::do_process(HashTableType& hash need_null_map_for_probe && ignore_null > (hash_table_ctx.keys, hash_table_ctx.bucket_nums.data(), probe_index, build_index, probe_rows, _probe_indexs.data(), - _probe_visited, _build_indexs.data()); + _probe_visited, _build_indexs.data(), has_mark_join_conjunct); Review Comment: need some regression test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](jdbc catalog) split sap_hana jdbc executor [doris]
doris-robot commented on PR #34733: URL: https://github.com/apache/doris/pull/34733#issuecomment-2105935386 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [refactor](jdbc catalog) split sap_hana jdbc executor [doris]
zy-kkk opened a new pull request, #34733: URL: https://github.com/apache/doris/pull/34733 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) improve spill log printing [doris]
yiguolei commented on code in PR #34729:
URL: https://github.com/apache/doris/pull/34729#discussion_r1597466368
##
be/src/runtime/workload_group/workload_group_manager.cpp:
##
@@ -201,7 +201,7 @@ void WorkloadGroupMgr::refresh_wg_memory_info() {
PrettyPrinter::print(process_mem_used, TUnit::BYTES),
PrettyPrinter::print(sys_mem_available, TUnit::BYTES),
PrettyPrinter::print(all_queries_mem_used, TUnit::BYTES));
-VLOG_EVERY_N(1, 10) << debug_msg;
+LOG_EVERY_T(INFO, 1) << debug_msg;
Review Comment:
very 10 seconds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) improve spill log printing [doris]
yiguolei commented on code in PR #34729:
URL: https://github.com/apache/doris/pull/34729#discussion_r1597466292
##
be/src/pipeline/pipeline_task.cpp:
##
@@ -315,7 +315,8 @@ bool PipelineTask::should_revoke_memory(RuntimeState*
state, int64_t revocable_m
wg->check_mem_used(_wg_mem_low_water_mark, _wg_mem_high_water_mark);
if (is_wg_mem_high_water_mark) {
if (revocable_mem_bytes > min_revocable_mem_bytes) {
-LOG_EVERY_N(INFO, 10) << "revoke memory, hight water mark";
+LOG_EVERY_T(INFO, 1) << "query " << print_id(state->query_id())
Review Comment:
Not print this log, it should be a debug log.
This method is called by pipeline engine during every loop.
If there is many queries, the log is too often.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](decimal) improve overflow error message [doris]
github-actions[bot] commented on PR #34689: URL: https://github.com/apache/doris/pull/34689#issuecomment-2105934203 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [Fix](executor)Fix workload thread start failed when follower convert to master
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new ad0e64bec17 [Fix](executor)Fix workload thread start failed when
follower convert to master
ad0e64bec17 is described below
commit ad0e64bec1749756277b20eb3f2bab78231eae8a
Author: wangbo
AuthorDate: Sat May 11 23:31:59 2024 +0800
[Fix](executor)Fix workload thread start failed when follower convert to
master
---
.../main/java/org/apache/doris/catalog/Env.java| 2 +-
.../resource/workloadgroup/WorkloadGroupMgr.java | 27 +++
.../WorkloadRuntimeStatusMgr.java | 66 +++-
.../WorkloadSchedPolicyMgr.java| 88 ++
.../workloadgroup/WorkloadGroupMgrTest.java| 33
.../data/workload_manager_p0/test_curd_wlg.out | 9 +++
regression-test/pipeline/p0/conf/fe.conf | 1 +
.../workload_manager_p0/test_curd_wlg.groovy | 6 ++
8 files changed, 91 insertions(+), 141 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index 0aa0da8ef36..4839769e0f8 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -1711,7 +1711,7 @@ public class Env {
dnsCache.start();
-workloadGroupMgr.startUpdateThread();
+workloadGroupMgr.start();
workloadSchedPolicyMgr.start();
workloadRuntimeStatusMgr.start();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
index a06413a94d4..2cfc59dfd2f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
@@ -34,6 +34,7 @@ import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.common.proc.BaseProcResult;
import org.apache.doris.common.proc.ProcResult;
+import org.apache.doris.common.util.MasterDaemon;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.persist.DropWorkloadGroupOperationLog;
import org.apache.doris.persist.gson.GsonPostProcessable;
@@ -64,7 +65,7 @@ import java.util.Map;
import java.util.Set;
import java.util.concurrent.locks.ReentrantReadWriteLock;
-public class WorkloadGroupMgr implements Writable, GsonPostProcessable {
+public class WorkloadGroupMgr extends MasterDaemon implements Writable,
GsonPostProcessable {
public static final String DEFAULT_GROUP_NAME = "normal";
@@ -90,22 +91,13 @@ public class WorkloadGroupMgr implements Writable,
GsonPostProcessable {
private final ResourceProcNode procNode = new ResourceProcNode();
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
-private Thread updatePropThread;
-
-public void startUpdateThread() {
-WorkloadGroupMgr wgMgr = this;
-updatePropThread = new Thread(() -> {
-Thread.currentThread().setName("reset-query-queue-prop");
-while (true) {
-try {
-wgMgr.resetQueryQueueProp();
-Thread.sleep(Config.query_queue_update_interval_ms);
-} catch (Throwable e) {
-LOG.warn("reset query queue failed ", e);
-}
-}
-});
-updatePropThread.start();
+@Override
+protected void runAfterCatalogReady() {
+try {
+resetQueryQueueProp();
+} catch (Throwable e) {
+LOG.warn("reset query queue failed ", e);
+}
}
public void resetQueryQueueProp() {
@@ -142,6 +134,7 @@ public class WorkloadGroupMgr implements Writable,
GsonPostProcessable {
}
public WorkloadGroupMgr() {
+super("workload-group-thread", Config.query_queue_update_interval_ms);
// if no fe image exist, we should append internal group here.
appendInternalWorkloadGroup();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
index ff2641d5f3c..06a58b295e5 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadRuntimeStatusMgr.java
@@ -19,7 +19,7 @@ package org.apache.doris.resource.workloadschedpolicy;
import org.apache.doris.catalog.Env;
import org.apache.doris.common.Config;
-import
Re: [PR] pick #34695 [doris]
yiguolei merged PR #34726: URL: https://github.com/apache/doris/pull/34726 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) Avoid unnecessary copies in the profile thrift. [doris]
Mryange commented on PR #34720: URL: https://github.com/apache/doris/pull/34720#issuecomment-2105933210 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](decimal) improve overflow error message [doris]
jacktengg commented on PR #34689: URL: https://github.com/apache/doris/pull/34689#issuecomment-2105932582 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) fuzzy spill and fix bugs [doris]
jacktengg commented on PR #33291: URL: https://github.com/apache/doris/pull/33291#issuecomment-2105932283 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) improve spill log printing [doris]
jacktengg commented on PR #34729: URL: https://github.com/apache/doris/pull/34729#issuecomment-2105931665 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) improve spill log printing [doris]
github-actions[bot] commented on PR #34729: URL: https://github.com/apache/doris/pull/34729#issuecomment-2105931373 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
