Re: [PR] [feature](backend) Add cpu core count and memory usage to "show backends" command. [doris]
HappenLee commented on code in PR #33885:
URL: https://github.com/apache/doris/pull/33885#discussion_r1571922363
##
be/src/agent/heartbeat_server.cpp:
##
@@ -88,11 +89,13 @@ void HeartbeatServer::heartbeat(THeartbeatResult&
heartbeat_result,
get_fragment_executing_count());
heartbeat_result.backend_info.__set_fragment_last_active_time(
get_fragment_last_active_time());
+heartbeat_result.backend_info.__set_be_mem(PerfCounters::get_vm_rss());
}
watch.stop();
if (watch.elapsed_time() > 1000L * 1000L * 1000L) {
LOG(WARNING) << "heartbeat consume too much time. time=" <<
watch.elapsed_time();
}
+LOG_WARNING("yxc test").tag("mem byte", PerfCounters::get_vm_rss());
Review Comment:
remove the test log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 09/14: [fix](multi-table-load) fix single stream multi table load cannot finish (#33816)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e38d844d40395f102da8dfef76c583ea5e29046e
Author: HHoflittlefish777 <77738092+hhoflittlefish...@users.noreply.github.com>
AuthorDate: Fri Apr 19 13:02:45 2024 +0800
[fix](multi-table-load) fix single stream multi table load cannot finish
(#33816)
---
be/src/io/fs/multi_table_pipe.cpp | 22 +-
be/src/io/fs/stream_load_pipe.cpp | 9 +
be/src/io/fs/stream_load_pipe.h | 4
3 files changed, 30 insertions(+), 5 deletions(-)
diff --git a/be/src/io/fs/multi_table_pipe.cpp
b/be/src/io/fs/multi_table_pipe.cpp
index 816561cfbc8..6a4da0188dd 100644
--- a/be/src/io/fs/multi_table_pipe.cpp
+++ b/be/src/io/fs/multi_table_pipe.cpp
@@ -114,17 +114,25 @@ Status MultiTablePipe::dispatch(const std::string& table,
const char* data, size
} else {
pipe = iter->second;
}
-RETURN_NOT_OK_STATUS_WITH_WARN((pipe.get()->*cb)(data, size),
- "append failed in unplanned kafka
pipe");
+// It is necessary to determine whether the sum of
pipe_current_capacity and size is greater than pipe_max_capacity,
+// otherwise the following situation may occur:
+// the pipe is full but still cannot trigger the request and exec plan
condition,
+// causing one stream multi table load can not finish
++_unplanned_row_cnt;
+auto pipe_current_capacity = pipe->current_capacity();
+auto pipe_max_capacity = pipe->max_capacity();
if (_unplanned_row_cnt >= _row_threshold ||
-_unplanned_pipes.size() >= _wait_tables_threshold) {
+_unplanned_pipes.size() >= _wait_tables_threshold ||
+pipe_current_capacity + size > pipe_max_capacity) {
LOG(INFO) << fmt::format(
"unplanned row cnt={} reach row_threshold={}
or "
- "wait_plan_table_threshold={}, "
+ "wait_plan_table_threshold={}, or the sum of "
+ "pipe_current_capacity {} "
+ "and size {} is greater than
pipe_max_capacity {}, "
"plan them",
- _unplanned_row_cnt, _row_threshold,
_wait_tables_threshold)
+ _unplanned_row_cnt, _row_threshold,
_wait_tables_threshold,
+ pipe_current_capacity, size,
pipe_max_capacity)
<< ", ctx: " << _ctx->brief();
Status st = request_and_exec_plans();
_unplanned_row_cnt = 0;
@@ -132,7 +140,11 @@ Status MultiTablePipe::dispatch(const std::string& table,
const char* data, size
return st;
}
}
+
+RETURN_NOT_OK_STATUS_WITH_WARN((pipe.get()->*cb)(data, size),
+ "append failed in unplanned kafka
pipe");
}
+
return Status::OK();
}
diff --git a/be/src/io/fs/stream_load_pipe.cpp
b/be/src/io/fs/stream_load_pipe.cpp
index cd5ee5a8a09..ecce306bdf1 100644
--- a/be/src/io/fs/stream_load_pipe.cpp
+++ b/be/src/io/fs/stream_load_pipe.cpp
@@ -255,5 +255,14 @@ TUniqueId StreamLoadPipe::calculate_pipe_id(const
UniqueId& query_id, int32_t fr
return pipe_id;
}
+size_t StreamLoadPipe::current_capacity() {
+std::unique_lock l(_lock);
+if (_use_proto) {
+return _proto_buffered_bytes;
+} else {
+return _buffered_bytes;
+}
+}
+
} // namespace io
} // namespace doris
diff --git a/be/src/io/fs/stream_load_pipe.h b/be/src/io/fs/stream_load_pipe.h
index 06c9bca4027..978badf9add 100644
--- a/be/src/io/fs/stream_load_pipe.h
+++ b/be/src/io/fs/stream_load_pipe.h
@@ -83,6 +83,10 @@ public:
// used for pipeline load, which use TUniqueId(lo: query_id.lo +
fragment_id, hi: query_id.hi) as pipe_id
static TUniqueId calculate_pipe_id(const UniqueId& query_id, int32_t
fragment_id);
+size_t max_capacity() const { return _max_buffered_bytes; }
+
+size_t current_capacity();
+
protected:
Status read_at_impl(size_t offset, Slice result, size_t* bytes_read,
const IOContext* io_ctx) override;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 11/14: [Enhancement](group commit) Add fault injection case for group commit
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 8b061c70551c7a4381c3568c3ca55c9a5f97ca86
Author: abmdocrt
AuthorDate: Fri Apr 19 13:05:19 2024 +0800
[Enhancement](group commit) Add fault injection case for group commit
---
be/src/runtime/group_commit_mgr.cpp| 11 +-
...oup_commit_async_wal_msg_fault_injection.groovy | 40 +-
2 files changed, 49 insertions(+), 2 deletions(-)
diff --git a/be/src/runtime/group_commit_mgr.cpp
b/be/src/runtime/group_commit_mgr.cpp
index 6d8873602d3..38e599180e7 100644
--- a/be/src/runtime/group_commit_mgr.cpp
+++ b/be/src/runtime/group_commit_mgr.cpp
@@ -333,6 +333,8 @@ Status GroupCommitTable::_finish_group_commit_load(int64_t
db_id, int64_t table_
RuntimeState* state) {
Status st;
Status result_status;
+DBUG_EXECUTE_IF("LoadBlockQueue._finish_group_commit_load.err_status",
+{ status = Status::InternalError(""); });
if (status.ok()) {
// commit txn
TLoadTxnCommitRequest request;
@@ -368,6 +370,13 @@ Status GroupCommitTable::_finish_group_commit_load(int64_t
db_id, int64_t table_
},
1L);
result_status = Status::create(result.status);
+DBUG_EXECUTE_IF("LoadBlockQueue._finish_group_commit_load.err_status",
{
+std ::string msg = "abort txn";
+LOG(INFO) << "debug promise set: " << msg;
+
ExecEnv::GetInstance()->group_commit_mgr()->debug_promise.set_value(
+Status ::InternalError(msg));
+return status;
+});
}
std::shared_ptr load_block_queue;
{
@@ -392,7 +401,7 @@ Status GroupCommitTable::_finish_group_commit_load(int64_t
db_id, int64_t table_
// status: exec_plan_fragment result
// st: commit txn rpc status
// result_status: commit txn result
-DBUG_EXECUTE_IF("LoadBlockQueue._finish_group_commit_load.err_status",
+DBUG_EXECUTE_IF("LoadBlockQueue._finish_group_commit_load.err_st",
{ st = Status::InternalError(""); });
if (status.ok() && st.ok() &&
(result_status.ok() ||
result_status.is())) {
diff --git
a/regression-test/suites/fault_injection_p0/test_group_commit_async_wal_msg_fault_injection.groovy
b/regression-test/suites/fault_injection_p0/test_group_commit_async_wal_msg_fault_injection.groovy
index cdf537749cc..c2523c49092 100644
---
a/regression-test/suites/fault_injection_p0/test_group_commit_async_wal_msg_fault_injection.groovy
+++
b/regression-test/suites/fault_injection_p0/test_group_commit_async_wal_msg_fault_injection.groovy
@@ -74,7 +74,7 @@
suite("test_group_commit_async_wal_msg_fault_injection","nonConcurrent") {
exception = false;
try {
GetDebugPoint().enableDebugPointForAllBEs("LoadBlockQueue._finish_group_commit_load.get_wal_back_pressure_msg")
-
GetDebugPoint().enableDebugPointForAllBEs("LoadBlockQueue._finish_group_commit_load.err_status")
+
GetDebugPoint().enableDebugPointForAllBEs("LoadBlockQueue._finish_group_commit_load.err_st")
streamLoad {
table "${tableName}"
set 'column_separator', ','
@@ -88,6 +88,44 @@
suite("test_group_commit_async_wal_msg_fault_injection","nonConcurrent") {
logger.info(e.getMessage())
assertTrue(e.getMessage().contains('estimated wal bytes 0 Bytes'))
exception = true;
+} finally {
+
GetDebugPoint().disableDebugPointForAllBEs("LoadBlockQueue._finish_group_commit_load.get_wal_back_pressure_msg")
+
GetDebugPoint().disableDebugPointForAllBEs("LoadBlockQueue._finish_group_commit_load.err_st")
+assertTrue(exception)
+}
+
+// test group commit abort txn
+sql """ DROP TABLE IF EXISTS ${tableName} """
+
+sql """
+CREATE TABLE IF NOT EXISTS ${tableName} (
+`k` int ,
+`v` int ,
+) engine=olap
+DISTRIBUTED BY HASH(`k`)
+BUCKETS 5
+properties("replication_num" = "1")
+"""
+
+GetDebugPoint().clearDebugPointsForAllBEs()
+
+exception = false;
+try {
+
GetDebugPoint().enableDebugPointForAllBEs("LoadBlockQueue._finish_group_commit_load.get_wal_back_pressure_msg")
+
GetDebugPoint().enableDebugPointForAllBEs("LoadBlockQueue._finish_group_commit_load.err_status")
+streamLoad {
+table "${tableName}"
+set 'column_separator', ','
+set 'group_commit', 'async_mode'
+unset 'label'
+file 'group_commit_wal_msg.csv'
+time 1
+}
+assertFalse(true);
+} catch (Exception e) {
+logger.info(e.getMessage(
(doris) 02/14: [enhancement](partial-update) print more log while missed some rowsets (#33711)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 56eb5ea00c5daf435ace18b06ea9d2ec934cb483
Author: zhannngchen <48427519+zhannngc...@users.noreply.github.com>
AuthorDate: Thu Apr 18 20:40:06 2024 +0800
[enhancement](partial-update) print more log while missed some rowsets
(#33711)
---
be/src/olap/rowset/segment_v2/segment_writer.cpp | 15 ---
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp | 15 ---
2 files changed, 16 insertions(+), 14 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/segment_writer.cpp
b/be/src/olap/rowset/segment_v2/segment_writer.cpp
index 284e0e0eaaa..377ffd6a56c 100644
--- a/be/src/olap/rowset/segment_v2/segment_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_writer.cpp
@@ -455,19 +455,20 @@ Status
SegmentWriter::append_block_with_partial_content(const vectorized::Block*
!_opts.rowset_ctx->partial_update_info->can_insert_new_rows_in_partial_update;
specified_rowsets =
tablet->get_rowset_by_ids(&_mow_context->rowset_ids,
should_include_stale);
-if (_opts.rowset_ctx->partial_update_info->is_strict_mode &&
-specified_rowsets.size() != _mow_context->rowset_ids.size()) {
+if (specified_rowsets.size() != _mow_context->rowset_ids.size()) {
// Only when this is a strict mode partial update that missing
rowsets here will lead to problems.
// In other case, the missing rowsets will be calculated in later
phases(commit phase/publish phase)
LOG(WARNING) << fmt::format(
"[Memtable Flush] some rowsets have been deleted due to "
-"compaction(specified_rowsets.size()={}, but
rowset_ids.size()={}) in strict "
-"mode partial update. tablet_id: {}, cur max_version: {},
transaction_id: {}",
+"compaction(specified_rowsets.size()={}, but
rowset_ids.size()={}) in "
+"partial update. tablet_id: {}, cur max_version: {},
transaction_id: {}",
specified_rowsets.size(), _mow_context->rowset_ids.size(),
_tablet->tablet_id(),
_mow_context->max_version, _mow_context->txn_id);
-return Status::InternalError(
-"[Memtable Flush] some rowsets have been deleted due to "
-"compaction in strict mode partial update");
+if (_opts.rowset_ctx->partial_update_info->is_strict_mode) {
+return Status::InternalError(
+"[Memtable Flush] some rowsets have been deleted due
to "
+"compaction in strict mode partial update");
+}
}
}
std::vector>
segment_caches(specified_rowsets.size());
diff --git a/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp
b/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp
index 2a4f924b98a..faa2ccdd321 100644
--- a/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp
@@ -387,19 +387,20 @@ Status
VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& da
!_opts.rowset_ctx->partial_update_info->can_insert_new_rows_in_partial_update;
specified_rowsets =
tablet->get_rowset_by_ids(&_mow_context->rowset_ids,
should_include_stale);
-if (_opts.rowset_ctx->partial_update_info->is_strict_mode &&
-specified_rowsets.size() != _mow_context->rowset_ids.size()) {
+if (specified_rowsets.size() != _mow_context->rowset_ids.size()) {
// Only when this is a strict mode partial update that missing
rowsets here will lead to problems.
// In other case, the missing rowsets will be calculated in later
phases(commit phase/publish phase)
LOG(WARNING) << fmt::format(
"[Memtable Flush] some rowsets have been deleted due to "
-"compaction(specified_rowsets.size()={}, but
rowset_ids.size()={}) in strict "
-"mode partial update. tablet_id: {}, cur max_version: {},
transaction_id: {}",
+"compaction(specified_rowsets.size()={}, but
rowset_ids.size()={}) in "
+"partial update. tablet_id: {}, cur max_version: {},
transaction_id: {}",
specified_rowsets.size(), _mow_context->rowset_ids.size(),
_tablet->tablet_id(),
_mow_context->max_version, _mow_context->txn_id);
-return Status::InternalError(
-"[Memtable Flush] some rowsets have been deleted due to "
-"compaction in strict mode partial update");
+if (_opts.rowset_ctx->partial_update_info->is_strict_mode) {
+return Status::InternalError(
+
(doris) 08/14: [Fix](inverted index) fix wrong need read data opt when encounters columnA > columnB predicate (#33855)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 659900040f7717b99b557391edb1509f3ae0d815
Author: airborne12
AuthorDate: Fri Apr 19 12:06:14 2024 +0800
[Fix](inverted index) fix wrong need read data opt when encounters columnA
> columnB predicate (#33855)
---
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 11 +-
.../test_index_no_need_read_data.out | 74 ++
.../test_index_no_need_read_data.groovy| 871 -
3 files changed, 954 insertions(+), 2 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index bee83282c48..ccdc0577a6b 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -2613,7 +2613,16 @@ void
SegmentIterator::_calculate_pred_in_remaining_conjunct_root(
auto node_type = expr->node_type();
if (node_type == TExprNodeType::SLOT_REF) {
-_column_predicate_info->column_name = expr->expr_name();
+if (_column_predicate_info->column_name.empty()) {
+_column_predicate_info->column_name = expr->expr_name();
+} else {
+// If column name already exists, create a new ColumnPredicateInfo
+// if expr is columnA > columnB, then column name will exist, in
this situation, we need to add it to _column_pred_in_remaining_vconjunct
+auto new_column_pred_info =
std::make_shared();
+new_column_pred_info->column_name = expr->expr_name();
+
_column_pred_in_remaining_vconjunct[new_column_pred_info->column_name].push_back(
+*new_column_pred_info);
+}
} else if (_is_literal_node(node_type)) {
auto v_literal_expr = static_cast(expr.get());
_column_predicate_info->query_value = v_literal_expr->value();
diff --git
a/regression-test/data/inverted_index_p0/test_index_no_need_read_data.out
b/regression-test/data/inverted_index_p0/test_index_no_need_read_data.out
index 9b64c110139..da7e5a81123 100644
--- a/regression-test/data/inverted_index_p0/test_index_no_need_read_data.out
+++ b/regression-test/data/inverted_index_p0/test_index_no_need_read_data.out
@@ -136,3 +136,77 @@ j 2 k
k 2 j
u 8 d
+-- !select_no_need_1 --
+1 -59609273357054098325 3
+1 -10 -10 9
+1 -10 -4 -4
+1 -10 1 2
+1 -10 3 6
+1 -10 4 -4
+1 -10 7 3
+1 -10 9 1
+2 -4 -10 -10
+1 -4 -10 3
+1 -4 3 3
+1 0 -4 5
+1 0 2 9
+1 0 5 1
+1 0 5 3
+1 1 -10 5
+1 1 0 -4
+1 1 4 3
+1 1 7 -10
+1 1 9 -4
+1 2 8 -4
+1 3 -10 8
+1 3 -4 9
+1 3 3 -4
+1 5 -10 -10
+1 5 -4 1
+1 5 1 8
+1 6 -10 -4
+1 7 -10 2
+1 7 1 6
+1 7 2 8
+1 7 3 -4
+1 7 8 -10
+1 8 -10 -10
+1 8 -10 3
+1 8 -10 8
+2 8 -10 9
+1 8 -4 5
+1 8 -4 8
+1 8 4 -4
+1 8 4 3
+1 8 5 -10
+1 8 5 2
+1 8 8 1
+1 9 -4 2
+1 9 6 1
+1 25 2 -10
+1 26 2 -10
+
+-- !select_no_need_2 --
+1.2
+
+-- !select_no_need_3 --
+2023-12-09 -55222131122733705612023-12-16
+2023-12-09 -41551379141288552352023-12-12
+2023-12-10 \N 2023-12-19
+2023-12-10 8625368487501035525 2023-12-17
+2023-12-11 4714785027515578912 2023-12-10
+2023-12-15 \N 2024-01-09
+2023-12-15 -559291350287121968 2023-12-16
+2023-12-16 -44734416847954620992023-12-15
+2023-12-18 \N 2025-06-18
+2023-12-18 419678391523752890 2026-01-18
+2023-12-20 6152093303748835632 2023-12-17
+2024-01-09 \N 2025-06-18
+2024-01-17 2371697304713448324 2024-01-09
+2025-02-17 -45992004400354582762024-01-19
+2025-06-18 -11963858744780929712023-12-17
+2025-06-18 -750688512374970345 2024-01-17
+2025-06-18 7670879000600913113 2023-12-09
+2026-01-18 9039465042666792521 2023-12-09
+2026-02-18 -39743429076950189102024-01-09
+
diff --git
a/regression-test/suites/inverted_index_p0/test_index_no_need_read_data.groovy
b/regression-test/suites/inverted_index_p0/test_index_no_need_read_data.groovy
index 31016df9814..fd871b0a000 100644
---
a/regression-test/suites/inverted_index_p0/test_index_no_need_read_
(doris) 01/14: [fix](txn insert) Fix txn insert commit failed when schema change (#33706)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5abc84af715456b882201b3f10a0594448418671
Author: meiyi
AuthorDate: Thu Apr 18 20:37:55 2024 +0800
[fix](txn insert) Fix txn insert commit failed when schema change (#33706)
---
.../apache/doris/planner/StreamLoadPlanner.java| 4 +
.../apache/doris/qe/InsertStreamTxnExecutor.java | 153 ++---
.../txn_insert_values_with_schema_change.out | 13 ++
.../txn_insert_values_with_schema_change.groovy| 109 +++
4 files changed, 202 insertions(+), 77 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/StreamLoadPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/StreamLoadPlanner.java
index 1e377389b44..8f75f5476f8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/StreamLoadPlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/StreamLoadPlanner.java
@@ -111,10 +111,12 @@ public class StreamLoadPlanner {
return destTable;
}
+// the caller should get table read lock when call this method
public TExecPlanFragmentParams plan(TUniqueId loadId) throws UserException
{
return this.plan(loadId, 1);
}
+// the caller should get table read lock when call this method
// create the plan. the plan's query id and load id are same, using the
parameter 'loadId'
public TExecPlanFragmentParams plan(TUniqueId loadId, int
fragmentInstanceIdIndex) throws UserException {
if (destTable.getKeysType() != KeysType.UNIQUE_KEYS
@@ -340,11 +342,13 @@ public class StreamLoadPlanner {
return params;
}
+// the caller should get table read lock when call this method
// single table plan fragmentInstanceIndex is 1(default value)
public TPipelineFragmentParams planForPipeline(TUniqueId loadId) throws
UserException {
return this.planForPipeline(loadId, 1);
}
+// the caller should get table read lock when call this method
public TPipelineFragmentParams planForPipeline(TUniqueId loadId, int
fragmentInstanceIdIndex) throws UserException {
if (destTable.getKeysType() != KeysType.UNIQUE_KEYS
&& taskInfo.getMergeType() != LoadTask.MergeType.APPEND) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/qe/InsertStreamTxnExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/InsertStreamTxnExecutor.java
index f37457cf58d..e89553e672d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/InsertStreamTxnExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/InsertStreamTxnExecutor.java
@@ -43,6 +43,7 @@ import org.apache.doris.thrift.TStreamLoadPutRequest;
import org.apache.doris.thrift.TTxnParams;
import org.apache.doris.thrift.TUniqueId;
import org.apache.doris.transaction.TransactionEntry;
+import org.apache.doris.transaction.TransactionState;
import org.apache.thrift.TException;
@@ -65,92 +66,90 @@ public class InsertStreamTxnExecutor {
public void beginTransaction(TStreamLoadPutRequest request) throws
UserException, TException, TimeoutException,
InterruptedException, ExecutionException {
TTxnParams txnConf = txnEntry.getTxnConf();
+OlapTable table = (OlapTable) txnEntry.getTable();
// StreamLoadTask's id == request's load_id
StreamLoadTask streamLoadTask =
StreamLoadTask.fromTStreamLoadPutRequest(request);
-StreamLoadPlanner planner = new StreamLoadPlanner(
-(Database) txnEntry.getDb(), (OlapTable) txnEntry.getTable(),
streamLoadTask);
-// Will using load id as query id in fragment
-if (Config.enable_pipeline_load) {
-TPipelineFragmentParams tRequest =
planner.planForPipeline(streamLoadTask.getId());
-BeSelectionPolicy policy = new
BeSelectionPolicy.Builder().needLoadAvailable().needQueryAvailable().build();
-List beIds =
Env.getCurrentSystemInfo().selectBackendIdsByPolicy(policy, 1);
-if (beIds.isEmpty()) {
-throw new UserException("No available backend to match the
policy: " + policy);
-}
-
-tRequest.setTxnConf(txnConf).setImportLabel(txnEntry.getLabel());
-for (Map.Entry> entry :
tRequest.local_params.get(0)
-.per_node_scan_ranges.entrySet()) {
-for (TScanRangeParams scanRangeParams : entry.getValue()) {
-
scanRangeParams.scan_range.ext_scan_range.file_scan_range.params.setFormatType(
-TFileFormatType.FORMAT_PROTO);
-
scanRangeParams.scan_range.ext_scan_range.file_scan_range.params.setCompressType(
-TFileCompressType.PLAIN);
+StreamLoadPlanner planner = new StreamLoadPlanner((Database)
txnEntry.getDb(), table, streamLoadTask);
+boolea
(doris) 07/14: [bugfix](paimon) Create paimon catalog with hadoop user (#33833)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1a6f8c443e329db2a4828c12fd815447369014b5
Author: wuwenchi
AuthorDate: Fri Apr 19 11:23:19 2024 +0800
[bugfix](paimon) Create paimon catalog with hadoop user (#33833)
When creating a catalog, paimon will create a warehouse on HDFS, so we need
to use the corresponding user with permissions to create it.
---
.../doris/datasource/paimon/PaimonExternalCatalog.java | 16 +---
1 file changed, 9 insertions(+), 7 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java
index 0a273e5424c..11691224681 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java
@@ -120,13 +120,15 @@ public abstract class PaimonExternalCatalog extends
ExternalCatalog {
}
protected Catalog createCatalog() {
-Options options = new Options();
-Map paimonOptionsMap = getPaimonOptionsMap();
-for (Map.Entry kv : paimonOptionsMap.entrySet()) {
-options.set(kv.getKey(), kv.getValue());
-}
-CatalogContext context = CatalogContext.create(options,
getConfiguration());
-return createCatalogImpl(context);
+return HadoopUGI.ugiDoAs(authConf, () -> {
+Options options = new Options();
+Map paimonOptionsMap = getPaimonOptionsMap();
+for (Map.Entry kv : paimonOptionsMap.entrySet()) {
+options.set(kv.getKey(), kv.getValue());
+}
+CatalogContext context = CatalogContext.create(options,
getConfiguration());
+return createCatalogImpl(context);
+});
}
protected Catalog createCatalogImpl(CatalogContext context) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 13/14: [improvement](storage) support glibc <2.21 for system call eventfd (#33218)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f4704b382196d872e4d41c8369851695453521bc
Author: On-Work-Song <165349831+on-work-s...@users.noreply.github.com>
AuthorDate: Fri Apr 19 14:27:28 2024 +0800
[improvement](storage) support glibc <2.21 for system call eventfd (#33218)
support glibc <2.21 for system call eventfd
---
be/src/glibc-compatibility/musl/eventfd.c | 8
1 file changed, 8 insertions(+)
diff --git a/be/src/glibc-compatibility/musl/eventfd.c
b/be/src/glibc-compatibility/musl/eventfd.c
index 68e489c8364..6358b921254 100644
--- a/be/src/glibc-compatibility/musl/eventfd.c
+++ b/be/src/glibc-compatibility/musl/eventfd.c
@@ -2,8 +2,16 @@
#include
#include
#include "syscall.h"
+#include
+
+
+
+#if __GLIBC__ > 2 || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 21)
int eventfd(unsigned int count, int flags)
+#else
+int eventfd(int count, int flags)
+#endif
{
int r = __syscall(SYS_eventfd2, count, flags);
#ifdef SYS_eventfd
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 03/14: [feature](variable) add read_only and super_read_only (#33795)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2675e94a9386071ce1c586282af49c5f37877364
Author: Mingyu Chen
AuthorDate: Thu Apr 18 23:56:19 2024 +0800
[feature](variable) add read_only and super_read_only (#33795)
---
.../java/org/apache/doris/qe/GlobalVariable.java | 14 -
.../data/variable_p0/set_and_unset_variable.out| 24 ++
.../variable_p0/set_and_unset_variable.groovy | 22
3 files changed, 59 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
index 30f91c6f8c1..15990e15ffe 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
@@ -57,6 +57,8 @@ public final class GlobalVariable {
public static final String AUDIT_PLUGIN_MAX_SQL_LENGTH =
"audit_plugin_max_sql_length";
public static final String ENABLE_GET_ROW_COUNT_FROM_FILE_LIST =
"enable_get_row_count_from_file_list";
+public static final String READ_ONLY = "read_only";
+public static final String SUPER_READ_ONLY = "super_read_only";
@VariableMgr.VarAttr(name = VERSION_COMMENT, flag = VariableMgr.READ_ONLY)
public static String versionComment = "Doris version "
@@ -135,7 +137,17 @@ public final class GlobalVariable {
+ "Getting file list may be a time-consuming
operation. "
+ "If you don't need to estimate the number of
rows in the table "
+ "or it affects performance, you can disable this
feature."})
-public static boolean enable_get_row_count_from_file_list = true;
+public static boolean enable_get_row_count_from_file_list = false;
+
+@VariableMgr.VarAttr(name = READ_ONLY, flag = VariableMgr.GLOBAL,
+description = {"仅用于兼容MySQL生态，暂无实际意义",
+"Only for compatibility with MySQL ecosystem, no practical
meaning"})
+public static boolean read_only = true;
+
+@VariableMgr.VarAttr(name = SUPER_READ_ONLY, flag = VariableMgr.GLOBAL,
+description = {"仅用于兼容MySQL生态，暂无实际意义",
+"Only for compatibility with MySQL ecosystem, no practical
meaning"})
+public static boolean super_read_only = true;
// Don't allow creating instance.
private GlobalVariable() {
diff --git a/regression-test/data/variable_p0/set_and_unset_variable.out
b/regression-test/data/variable_p0/set_and_unset_variable.out
index e7f0dc2fdf1..33dd8af7bc1 100644
--- a/regression-test/data/variable_p0/set_and_unset_variable.out
+++ b/regression-test/data/variable_p0/set_and_unset_variable.out
@@ -161,3 +161,27 @@ deprecated_enable_local_exchange truetrue0
-- !cmd --
show_hidden_columnsfalse false 0
+-- !cmd --
+read_only truetrue0
+
+-- !cmd --
+0
+
+-- !cmd --
+read_only truetrue0
+
+-- !cmd --
+read_only truetrue0
+
+-- !cmd --
+super_read_onlytruetrue0
+
+-- !cmd --
+0
+
+-- !cmd --
+super_read_onlytruetrue0
+
+-- !cmd --
+super_read_onlytruetrue0
+
diff --git a/regression-test/suites/variable_p0/set_and_unset_variable.groovy
b/regression-test/suites/variable_p0/set_and_unset_variable.groovy
index 47749774336..c637d3ae223 100644
--- a/regression-test/suites/variable_p0/set_and_unset_variable.groovy
+++ b/regression-test/suites/variable_p0/set_and_unset_variable.groovy
@@ -83,4 +83,26 @@ suite("set_and_unset_variable") {
qt_cmd """show global variables like 'experimental_enable_agg_state'"""
qt_cmd """show global variables like 'deprecated_enable_local_exchange'"""
qt_cmd """show global variables like 'show_hidden_columns'"""
+
+// test read_only
+qt_cmd """show variables like 'read_only'"""
+test {
+sql "set read_only=true"
+exception "should be set with SET GLOBAL"
+}
+qt_cmd "set global read_only=true"
+qt_cmd """show global variables like 'read_only'"""
+qt_cmd """show variables like 'read_only'"""
+sql "set global read_only=false"
+
+// test super_read_only
+qt_cmd """show variables like 'super_read_only'"""
+test {
+sql "set super_read_only=true"
+exception "should be set with SET GLOBAL"
+}
+qt_cmd "set global super_read_only=true"
+qt_cmd """show global variables like 'super_read_only'"""
+qt_cmd """show variables like 'super_read_only'"""
+sql "set global super_read_only=false"
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (315f6e44c2f -> c747714c186)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from 315f6e44c2f [Branch-2.1](Outfile) Fixed the problem that the concurrent Outfile wrote multiple Success files (#33870) new 5abc84af715 [fix](txn insert) Fix txn insert commit failed when schema change (#33706) new 56eb5ea00c5 [enhancement](partial-update) print more log while missed some rowsets (#33711) new 2675e94a938 [feature](variable) add read_only and super_read_only (#33795) new ffd9da44a2a [fix](move-memtable) fix commit may fail due to duplicated reports (#32403) new a8ba933947c [Fix](nereids) fix bind order by expression logic (#33843) new 6776a3ad1b9 [Fix](planner) fix create view star except and modify cast to sql (#33726) new 1a6f8c443e3 [bugfix](paimon) Create paimon catalog with hadoop user (#33833) new 659900040f7 [Fix](inverted index) fix wrong need read data opt when encounters columnA > columnB predicate (#33855) new e38d844d403 [fix](multi-table-load) fix single stream multi table load cannot finish (#33816) new ad75b9b1427 [opt](auto bucket) add fe config autobucket_max_buckets (#33842) new 8b061c70551 [Enhancement](group commit) Add fault injection case for group commit new 175e85d616e [Bug](runtime-filter) fix coredump on no null string type rf (#33869) new f4704b38219 [improvement](storage) support glibc <2.21 for system call eventfd (#33218) new c747714c186 [fix](memory) Fix ExecEnv destroy memory tracking (#33781) The 14 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/exprs/hybrid_set.h | 4 +- be/src/glibc-compatibility/musl/eventfd.c | 8 + be/src/io/fs/multi_table_pipe.cpp | 22 +- be/src/io/fs/stream_load_pipe.cpp | 9 + be/src/io/fs/stream_load_pipe.h| 4 + be/src/olap/rowset/segment_v2/segment_iterator.cpp | 11 +- be/src/olap/rowset/segment_v2/segment_writer.cpp | 15 +- .../rowset/segment_v2/vertical_segment_writer.cpp | 15 +- be/src/runtime/group_commit_mgr.cpp| 11 +- be/src/runtime/memory/mem_tracker_limiter.h| 4 +- be/src/runtime/thread_context.h| 13 +- be/src/vec/sink/load_stream_stub.h | 12 + be/src/vec/sink/writer/vtablet_writer_v2.cpp | 86 +- be/src/vec/sink/writer/vtablet_writer_v2.h | 7 + be/test/vec/sink/vtablet_writer_v2_test.cpp| 239 ++ .../java/org/apache/doris/catalog/ScalarType.java | 2 +- .../main/java/org/apache/doris/common/Config.java | 8 +- .../java/org/apache/doris/analysis/CastExpr.java | 2 +- .../java/org/apache/doris/analysis/SelectStmt.java | 7 +- .../apache/doris/common/util/AutoBucketUtils.java | 3 +- .../datasource/paimon/PaimonExternalCatalog.java | 16 +- .../nereids/rules/analysis/BindExpression.java | 4 +- .../apache/doris/planner/StreamLoadPlanner.java| 4 + .../java/org/apache/doris/qe/GlobalVariable.java | 14 +- .../apache/doris/qe/InsertStreamTxnExecutor.java | 153 ++-- .../analysis/CreateTableAsSelectStmtTest.java | 4 +- .../doris/planner/TableFunctionPlanTest.java | 2 +- .../txn_insert_values_with_schema_change.out | 13 + .../test_index_no_need_read_data.out | 74 ++ .../nereids_syntax_p0/order_by_bind_priority.out | 22 + .../join/no_null_str_rf/no_null_str_rf.out | 4 + .../data/variable_p0/set_and_unset_variable.out| 24 + .../create_view_star_except_and_cast_to_sql.out| 11 + .../suites/autobucket/test_autobucket.groovy | 33 +- .../test_commit_info_fault_injection.groovy| 99 +++ ...oup_commit_async_wal_msg_fault_injection.groovy | 40 +- .../txn_insert_values_with_schema_change.groovy| 109 +++ .../test_index_no_need_read_data.groovy| 871 - .../order_by_bind_priority.groovy | 36 + .../join/no_null_str_rf/no_null_str_rf.groovy | 58 ++ .../variable_p0/set_and_unset_variable.groovy | 22 + .../create_view_star_except_and_cast_to_sql.groovy | 45 ++ 42 files changed, 1976 insertions(+), 164 deletions(-) create mode 100644 be/test/vec/sink/vtablet_writer_v2_test.cpp create mode 100644 regression-test/data/insert_p0/txn_insert_values_with_schema_change.out create mode 100644 regression-test/data/query_p0/join/no_null_str_rf/no_null_str_rf.out create mode 100644 regression-test/data/view_p0/create_view_star_except_and_cast_to_sql.out create mode 100644 regression-test/suites/fault_injection_p0/test_commit_info_fault_injection.groovy create mode 100644 regression-test/s
(doris) 04/14: [fix](move-memtable) fix commit may fail due to duplicated reports (#32403)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ffd9da44a2a1803bcab1d2c4895eea9eed9f6e44
Author: Kaijie Chen
AuthorDate: Fri Apr 19 10:13:18 2024 +0800
[fix](move-memtable) fix commit may fail due to duplicated reports (#32403)
---
be/src/vec/sink/load_stream_stub.h | 12 ++
be/src/vec/sink/writer/vtablet_writer_v2.cpp | 86 +---
be/src/vec/sink/writer/vtablet_writer_v2.h | 7 +
be/test/vec/sink/vtablet_writer_v2_test.cpp| 239 +
.../test_commit_info_fault_injection.groovy| 99 +
5 files changed, 414 insertions(+), 29 deletions(-)
diff --git a/be/src/vec/sink/load_stream_stub.h
b/be/src/vec/sink/load_stream_stub.h
index 5ebec9f9d78..8ef40b84145 100644
--- a/be/src/vec/sink/load_stream_stub.h
+++ b/be/src/vec/sink/load_stream_stub.h
@@ -200,6 +200,18 @@ public:
std::string to_string();
+// for tests only
+void add_success_tablet(int64_t tablet_id) {
+std::lock_guard lock(_success_tablets_mutex);
+_success_tablets.push_back(tablet_id);
+}
+
+// for tests only
+void add_failed_tablet(int64_t tablet_id, Status reason) {
+std::lock_guard lock(_failed_tablets_mutex);
+_failed_tablets[tablet_id] = reason;
+}
+
private:
Status _encode_and_send(PStreamHeader& header, std::span data
= {});
Status _send_with_buffer(butil::IOBuf& buf, bool sync = false);
diff --git a/be/src/vec/sink/writer/vtablet_writer_v2.cpp
b/be/src/vec/sink/writer/vtablet_writer_v2.cpp
index 1f1756b5a16..21a87c150b8 100644
--- a/be/src/vec/sink/writer/vtablet_writer_v2.cpp
+++ b/be/src/vec/sink/writer/vtablet_writer_v2.cpp
@@ -574,39 +574,28 @@ Status VTabletWriterV2::close(Status exec_status) {
// calculate and submit commit info
if (is_last_sink) {
-std::unordered_map failed_tablets;
-std::unordered_map failed_reason;
-std::vector tablet_commit_infos;
-
-_load_stream_map->for_each([&](int64_t dst_id, const Streams&
streams) {
-std::unordered_set known_tablets;
-for (const auto& stream : streams) {
-for (auto [tablet_id, reason] : stream->failed_tablets()) {
-if (known_tablets.contains(tablet_id)) {
-continue;
-}
-known_tablets.insert(tablet_id);
-failed_tablets[tablet_id]++;
-failed_reason[tablet_id] = reason;
-}
-for (auto tablet_id : stream->success_tablets()) {
-if (known_tablets.contains(tablet_id)) {
-continue;
-}
-known_tablets.insert(tablet_id);
-TTabletCommitInfo commit_info;
-commit_info.tabletId = tablet_id;
-commit_info.backendId = dst_id;
-
tablet_commit_infos.emplace_back(std::move(commit_info));
+DBUG_EXECUTE_IF("VTabletWriterV2.close.add_failed_tablet", {
+auto streams =
_load_stream_map->at(_tablets_for_node.begin()->first);
+int64_t tablet_id = -1;
+for (auto& stream : *streams) {
+const auto& tablets = stream->success_tablets();
+if (tablets.size() > 0) {
+tablet_id = tablets[0];
+break;
}
}
+if (tablet_id != -1) {
+LOG(INFO) << "fault injection: adding failed tablet_id: "
<< tablet_id;
+streams->front()->add_failed_tablet(tablet_id,
+
Status::InternalError("fault injection"));
+} else {
+LOG(INFO) << "fault injection: failed to inject failed
tablet_id";
+}
});
-for (auto [tablet_id, replicas] : failed_tablets) {
-if (replicas > (_num_replicas - 1) / 2) {
-return failed_reason.at(tablet_id);
-}
-}
+std::vector tablet_commit_infos;
+RETURN_IF_ERROR(
+_create_commit_info(tablet_commit_infos, _load_stream_map,
_num_replicas));
_state->tablet_commit_infos().insert(
_state->tablet_commit_infos().end(),
std::make_move_iterator(tablet_commit_infos.begin()),
@@ -659,4 +648,43 @@ void VTabletWriterV2::_calc_tablets_to_commit() {
}
}
+Status VTabletWriterV2::_create_commit_info(std::vector&
tablet_commit_infos,
+std::shared_ptr
load_stream_map,
+
(doris) 05/14: [Fix](nereids) fix bind order by expression logic (#33843)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a8ba933947cb45fff9d6cdd86f5f6fc484463cbb
Author: feiniaofeiafei <53502832+feiniaofeia...@users.noreply.github.com>
AuthorDate: Fri Apr 19 10:43:26 2024 +0800
[Fix](nereids) fix bind order by expression logic (#33843)
---
.../nereids/rules/analysis/BindExpression.java | 4 +--
.../nereids_syntax_p0/order_by_bind_priority.out | 22 +
.../order_by_bind_priority.groovy | 36 ++
3 files changed, 60 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
index 8957800c7ed..3c73a53dcba 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
@@ -774,9 +774,9 @@ public class BindExpression implements AnalysisRuleFactory {
(self, unboundSlot) -> {
// first, try to bind slot in Scope(input.output)
List slotsInInput =
self.bindExactSlotsByThisScope(unboundSlot, inputScope);
-if (slotsInInput.size() == 1) {
+if (!slotsInInput.isEmpty()) {
// bind succeed
-return slotsInInput;
+return ImmutableList.of(slotsInInput.get(0));
}
// second, bind failed:
// if the slot not found, or more than one candidate slots
found in input.output,
diff --git a/regression-test/data/nereids_syntax_p0/order_by_bind_priority.out
b/regression-test/data/nereids_syntax_p0/order_by_bind_priority.out
index 3c7dcb1d318..4fd63bc7f99 100644
--- a/regression-test/data/nereids_syntax_p0/order_by_bind_priority.out
+++ b/regression-test/data/nereids_syntax_p0/order_by_bind_priority.out
@@ -23,3 +23,25 @@
3 3
5 5
+-- !test_multi_slots_in_agg_func_bind_first --
+\N -3 2 \N \N \N
+\N 0 5 \N 16155 16155
+\N 1 6 \N \N \N
+\N 2 7 \N \N \N
+\N 6 11 \N \N \N
+\N 8 13 \N \N \N
+\N 13 18 \N \N \N
+2 -5 0 8777\N \N
+2 -4 1 127 30240 30240
+2 -2 3 10008 -54 -54
+2 3 8 29433 -4654 -4654
+2 4 9 13909 29600 29600
+2 12 17 22950 99 99
+4 -1 4 361828812881
+4 5 10 10450 81058105
+4 7 12 88 88 88
+4 9 14 74 14138 14138
+4 10 15 23 63 63
+4 11 16 4418-24598 -24598
+4 14 19 2 \N \N
+
diff --git
a/regression-test/suites/nereids_syntax_p0/order_by_bind_priority.groovy
b/regression-test/suites/nereids_syntax_p0/order_by_bind_priority.groovy
index e434ab42092..44c8b5fa734 100644
--- a/regression-test/suites/nereids_syntax_p0/order_by_bind_priority.groovy
+++ b/regression-test/suites/nereids_syntax_p0/order_by_bind_priority.groovy
@@ -33,6 +33,42 @@ suite("order_by_bind_priority") {
sql "select abs(sum(c1)) as c1, c1,sum(c2) as c2 from
t_order_by_bind_priority group by c1 order by sum(c1)+c2 asc;"
exception "c2 should be grouped by."
}
+sql """drop table if exists
table_20_undef_partitions2_keys3_properties4_distributed_by58"""
+sql """
+create table
table_20_undef_partitions2_keys3_properties4_distributed_by58 (
+pk int,
+col_int_undef_signed int ,
+col_int_undef_signed2 int
+) engine=olap
+DUPLICATE KEY(pk, col_int_undef_signed)
+distributed by hash(pk) buckets 10
+properties("replication_num" = "1");
+"""
+sql """
+insert into
table_20_undef_partitions2_keys3_properties4_distributed_by58(pk,col_int_undef_signed,col_int_undef_signed2)
values (0,-8777,null),
+
(1,-127,30240),(2,null,null),(3,-10008,-54),(4,3618,2881),(5,null,16155),(6,null,null),(7,null,null),(8,-29433,-4654),(9,-13909,29600),(10,10450,8105),
+
(11,null,null),(12,88,88),(13,null,null),(14,74,14138),(15,23,63),(16,4418,-24598),(17,-22950,99),(18,null,null),(19,2,null);
+"""
+sql "sync;"
+qt_test_multi_slots_in_agg_func_bind_first """
+SELECT
+SIGN( SUM(col_int_undef_signed) ) + 3 AS col_int_undef_signed,
+pk - 5 pk ,
+pk pk ,
+ABS( MIN(col_int_undef_signed) ) AS col_int_undef_signed,
+MAX(col_int_undef_signed2) col_int_undef_signed2,
+col_int_undef_signed2 col_int_un
(doris) 12/14: [Bug](runtime-filter) fix coredump on no null string type rf (#33869)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 175e85d616ea3cbc01baebaf6942a97483070510
Author: Pxl
AuthorDate: Fri Apr 19 14:07:12 2024 +0800
[Bug](runtime-filter) fix coredump on no null string type rf (#33869)
fix coredump on no null string type rf
---
be/src/exprs/hybrid_set.h | 4 +-
.../join/no_null_str_rf/no_null_str_rf.out | 4 ++
.../join/no_null_str_rf/no_null_str_rf.groovy | 58 ++
3 files changed, 64 insertions(+), 2 deletions(-)
diff --git a/be/src/exprs/hybrid_set.h b/be/src/exprs/hybrid_set.h
index 03cde80c24b..c45167a83bb 100644
--- a/be/src/exprs/hybrid_set.h
+++ b/be/src/exprs/hybrid_set.h
@@ -421,7 +421,7 @@ public:
void _insert_fixed_len_string(const auto& col, const uint8_t* __restrict
nullmap, size_t start,
size_t end) {
for (size_t i = start; i < end; i++) {
-if (nullmap != nullptr || !nullmap[i]) {
+if (nullmap == nullptr || !nullmap[i]) {
_set.insert(col.get_data_at(i).to_string());
} else {
_contains_null = true;
@@ -583,7 +583,7 @@ public:
void _insert_fixed_len_string(const auto& col, const uint8_t* __restrict
nullmap, size_t start,
size_t end) {
for (size_t i = start; i < end; i++) {
-if (nullmap != nullptr || !nullmap[i]) {
+if (nullmap == nullptr || !nullmap[i]) {
_set.insert(col.get_data_at(i));
} else {
_contains_null = true;
diff --git
a/regression-test/data/query_p0/join/no_null_str_rf/no_null_str_rf.out
b/regression-test/data/query_p0/join/no_null_str_rf/no_null_str_rf.out
new file mode 100644
index 000..ebb4260eeb2
--- /dev/null
+++ b/regression-test/data/query_p0/join/no_null_str_rf/no_null_str_rf.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !test --
+4
+
diff --git
a/regression-test/suites/query_p0/join/no_null_str_rf/no_null_str_rf.groovy
b/regression-test/suites/query_p0/join/no_null_str_rf/no_null_str_rf.groovy
new file mode 100644
index 000..0b13db03a20
--- /dev/null
+++ b/regression-test/suites/query_p0/join/no_null_str_rf/no_null_str_rf.groovy
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("no_null_str_rf") {
+sql """ DROP TABLE IF EXISTS d_table; """
+sql """ DROP TABLE IF EXISTS dd_table; """
+sql """
+create table d_table(
+k1 int null,
+k2 int not null,
+k3 bigint null,
+k4 varchar(100) not null
+)
+duplicate key (k1,k2,k3)
+distributed BY hash(k1) buckets 3
+properties("replication_num" = "1");
+"""
+
+sql "insert into d_table select 1,1,1,'a';"
+sql "insert into d_table select 1,1,1,'b';"
+sql "insert into d_table select 1,1,1,'a';"
+sql "insert into d_table select 1,1,1,'b';"
+sql "insert into d_table select 1,1,1,'a';"
+sql "insert into d_table select 1,1,1,'b';"
+sql "insert into d_table select 1,1,1,'a';"
+sql "insert into d_table select 1,1,1,'b';"
+
+sql """
+create table dd_table(
+k1 int null,
+k2 int not null,
+k3 bigint null,
+k4 varchar(100) not null
+)
+duplicate key (k1,k2,k3)
+distributed BY hash(k1) buckets 3
+properties("replication_num" = "1");
+"""
+
+sql "insert into dd_table select 1,1,1,'a';"
+sql "insert into dd_table select 1,1,1,'c';"
+
+qt_test """select count(1) from d_table,dd_table where
d_table.k4=dd_table.k4 and d_table.k1=1 and dd_table.k1=1;"""
+}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 14/14: [fix](memory) Fix ExecEnv destroy memory tracking (#33781)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c747714c1864e08311bc3c76e3537453615c0cae
Author: Xinyi Zou
AuthorDate: Fri Apr 19 14:41:22 2024 +0800
[fix](memory) Fix ExecEnv destroy memory tracking (#33781)
disable memory tracking when ExecEnv destroy.
fix memory tracker label convert to query id
---
be/src/runtime/memory/mem_tracker_limiter.h | 4 ++--
be/src/runtime/thread_context.h | 13 +++--
2 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/be/src/runtime/memory/mem_tracker_limiter.h
b/be/src/runtime/memory/mem_tracker_limiter.h
index fcb319a8d77..240170ea881 100644
--- a/be/src/runtime/memory/mem_tracker_limiter.h
+++ b/be/src/runtime/memory/mem_tracker_limiter.h
@@ -217,8 +217,8 @@ public:
// only for Type::QUERY or Type::LOAD.
static TUniqueId label_to_queryid(const std::string& label) {
-if (label.rfind("Query#Id=", 0) != 0 && label.rfind("Load#Id=", 0) !=
0) {
-return TUniqueId();
+if (label.find("#Id=") == std::string::npos) {
+return {};
}
auto queryid = split(label, "#Id=")[1];
TUniqueId querytid;
diff --git a/be/src/runtime/thread_context.h b/be/src/runtime/thread_context.h
index 09721fb5d04..6920dfabe60 100644
--- a/be/src/runtime/thread_context.h
+++ b/be/src/runtime/thread_context.h
@@ -79,13 +79,14 @@
#define SCOPED_CONSUME_MEM_TRACKER_BY_HOOK(mem_tracker) \
auto VARNAME_LINENUM(add_mem_consumer) =
doris::AddThreadMemTrackerConsumerByHook(mem_tracker)
-#define ORPHAN_TRACKER_CHECK()
\
-DCHECK(!doris::config::enable_memory_orphan_check ||
\
- doris::thread_context()->thread_mem_tracker()->label() != "Orphan")
\
+#define ORPHAN_TRACKER_CHECK()
\
+DCHECK(doris::k_doris_exit || !doris::config::enable_memory_orphan_check
|| \
+ doris::thread_context()->thread_mem_tracker()->label() != "Orphan")
\
<< doris::memory_orphan_check_msg
-#define MEMORY_ORPHAN_CHECK() \
-DCHECK(!doris::config::enable_memory_orphan_check) <<
doris::memory_orphan_check_msg;
+#define MEMORY_ORPHAN_CHECK() \
+DCHECK(doris::k_doris_exit || !doris::config::enable_memory_orphan_check) \
+<< doris::memory_orphan_check_msg;
#else
#define SCOPED_MEM_COUNT_BY_HOOK(scope_mem) (void)0
#define SCOPED_CONSUME_MEM_TRACKER_BY_HOOK(mem_tracker) (void)0
@@ -200,7 +201,7 @@ public:
void consume_memory(const int64_t size) const {
#ifdef USE_MEM_TRACKER
-DCHECK(!doris::config::enable_memory_orphan_check ||
+DCHECK(doris::k_doris_exit ||
!doris::config::enable_memory_orphan_check ||
thread_mem_tracker()->label() != "Orphan")
<< doris::memory_orphan_check_msg;
#endif
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 10/14: [opt](auto bucket) add fe config autobucket_max_buckets (#33842)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ad75b9b1427eaa579fb3e22ef2eecc616ce02754
Author: Kang
AuthorDate: Fri Apr 19 13:04:21 2024 +0800
[opt](auto bucket) add fe config autobucket_max_buckets (#33842)
---
.../main/java/org/apache/doris/common/Config.java | 8 +-
.../apache/doris/common/util/AutoBucketUtils.java | 3 +-
.../suites/autobucket/test_autobucket.groovy | 33 +++---
3 files changed, 38 insertions(+), 6 deletions(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 8dff0e3f4db..3525d411b66 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -2341,12 +2341,18 @@ public class Config extends ConfigBase {
})
public static long analyze_record_limit = 2;
-@ConfField(description = {
+@ConfField(mutable = true, description = {
"Auto Buckets中最小的buckets数目",
"min buckets of auto bucket"
})
public static int autobucket_min_buckets = 1;
+@ConfField(mutable = true, description = {
+"Auto Buckets中最大的buckets数目",
+"max buckets of auto bucket"
+})
+public static int autobucket_max_buckets = 128;
+
@ConfField(description = {"Arrow Flight
Server中所有用户token的缓存上限，超过后LRU淘汰，默认值为512, "
+ "并强制限制小于 qe_max_connection/2, 避免`Reach limit of connections`, "
+ "因为arrow flight sql是无状态的协议，连接通常不会主动断开，"
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/AutoBucketUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/AutoBucketUtils.java
index 294250fd213..55a07b89eb8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/AutoBucketUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/AutoBucketUtils.java
@@ -20,6 +20,7 @@ package org.apache.doris.common.util;
import org.apache.doris.catalog.DiskInfo;
import org.apache.doris.catalog.DiskInfo.DiskState;
import org.apache.doris.catalog.Env;
+import org.apache.doris.common.Config;
import org.apache.doris.system.Backend;
import org.apache.doris.system.SystemInfoService;
@@ -85,7 +86,7 @@ public class AutoBucketUtils {
public static int getBucketsNum(long partitionSize) {
int bucketsNumByPartitionSize =
convertParitionSizeToBucketsNum(partitionSize);
int bucketsNumByBE = getBucketsNumByBEDisks();
-int bucketsNum = Math.min(128, Math.min(bucketsNumByPartitionSize,
bucketsNumByBE));
+int bucketsNum = Math.min(Config.autobucket_max_buckets,
Math.min(bucketsNumByPartitionSize, bucketsNumByBE));
int beNum = getBENum();
logger.debug("AutoBucketsUtil: bucketsNumByPartitionSize {},
bucketsNumByBE {}, bucketsNum {}, beNum {}",
bucketsNumByPartitionSize, bucketsNumByBE, bucketsNum, beNum);
diff --git a/regression-test/suites/autobucket/test_autobucket.groovy
b/regression-test/suites/autobucket/test_autobucket.groovy
index d3ba70d0df3..615ede5675b 100644
--- a/regression-test/suites/autobucket/test_autobucket.groovy
+++ b/regression-test/suites/autobucket/test_autobucket.groovy
@@ -40,7 +40,8 @@ suite("test_autobucket") {
sql "drop table if exists autobucket_test"
-
+// set min to 5
+sql "ADMIN SET FRONTEND CONFIG ('autobucket_min_buckets' = '5')"
sql "drop table if exists autobucket_test_min_buckets"
result = sql """
CREATE TABLE `autobucket_test_min_buckets` (
@@ -55,11 +56,35 @@ suite("test_autobucket") {
)
"""
-default_min_buckets = 1 // in Config.java
result = sql "show partitions from autobucket_test_min_buckets"
logger.info("${result}")
// XXX: buckets at pos(8), next maybe impl by sql meta
-assertEquals(Integer.valueOf(result.get(0).get(8)), default_min_buckets)
-
+assertEquals(Integer.valueOf(result.get(0).get(8)), 5)
+// set back to default
+sql "ADMIN SET FRONTEND CONFIG ('autobucket_min_buckets' = '1')"
sql "drop table if exists autobucket_test_min_buckets"
+
+// set max to 4
+sql "ADMIN SET FRONTEND CONFIG ('autobucket_max_buckets' = '4')"
+sql "drop table if exists autobucket_test_max_buckets"
+result = sql """
+CREATE TABLE `autobucket_test_max_buckets` (
+ `user_id` largeint(40) NOT NULL
+) ENGINE=OLAP
+DUPLICATE KEY(`user_id`)
+COMMENT 'OLAP'
+DISTRIBUTED BY HASH(`user_id`) BUCKETS AUTO
+PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "estimate_partition_size" = "10G"
+)
+"""
+
+result = sql "show partitions from autobucket_test_max_buckets"
+logger.info("${result}")
+// XXX: buckets at pos(8), next maybe impl by sq
Re: [PR] [fix](DECIMAL) error DECIMAL cat to BOOLEAN [doris]
zhiqiang- commented on PR #33868: URL: https://github.com/apache/doris/pull/33868#issuecomment-2065889177 https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html#function_cast cast function of mysql do not support using bool as type argument. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](backend) Add cpu core count and memory usage to "show backends" command. [doris]
Mryange commented on PR #33885: URL: https://github.com/apache/doris/pull/33885#issuecomment-2065891268 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] be自动退出 [doris]
KongFuPandaa commented on issue #33815: URL: https://github.com/apache/doris/issues/33815#issuecomment-2065892441 我也遇到了 be自动退出，然后还发现有内存溢出，java进程一直在累积内存，一天几百M到几十M增长量，开机四天从1.5G内存涨到了今日的5.4G 大概每日插入数据在万条左右 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](cloud)avoid transaction too large when dropping partition [doris]
xy720 commented on PR #33700: URL: https://github.com/apache/doris/pull/33700#issuecomment-2065892826 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] 执行查询出现内存不够 [doris]
KongFuPandaa commented on issue #33565: URL: https://github.com/apache/doris/issues/33565#issuecomment-2065893401 我也遇到了，感觉像是内存溢出的问题，希望解决也Q一下我 谢谢！！ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](backend) Add cpu core count and memory usage to "show backends" command. [doris]
HappenLee commented on code in PR #33885: URL: https://github.com/apache/doris/pull/33885#discussion_r1571930840 ## be/src/agent/heartbeat_server.cpp: ## @@ -88,6 +89,7 @@ void HeartbeatServer::heartbeat(THeartbeatResult& heartbeat_result, get_fragment_executing_count()); heartbeat_result.backend_info.__set_fragment_last_active_time( get_fragment_last_active_time()); +heartbeat_result.backend_info.__set_be_mem(PerfCounters::get_vm_rss()); Review Comment: use mem_info `_s_physical_mem` not this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [exec](table_fun) opt bitmap/split vexplode table func performance [doris]
yiguolei merged PR #33876: URL: https://github.com/apache/doris/pull/33876 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (3740b97164c -> 13c110863b5)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 3740b97164c [fix](memory) Fix ExecEnv destroy memory tracking (#33781) add 13c110863b5 [exec](table_fun) opt bitmap/split vexplode table func performance (#33876) No new revisions were added by this update. Summary of changes: .../vec/exprs/table_function/vexplode_bitmap.cpp | 28 ++ be/src/vec/exprs/table_function/vexplode_bitmap.h | 2 ++ be/src/vec/exprs/table_function/vexplode_split.cpp | 33 ++ be/src/vec/exprs/table_function/vexplode_split.h | 3 +- 4 files changed, 60 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](compile) Fix gcc compile on master [doris]
github-actions[bot] commented on PR #33864: URL: https://github.com/apache/doris/pull/33864#issuecomment-2065900878 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](compile) Fix gcc compile on master [doris]
github-actions[bot] commented on PR #33864: URL: https://github.com/apache/doris/pull/33864#issuecomment-2065900949 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](cloud)avoid transaction too large when dropping partition [doris]
github-actions[bot] commented on PR #33700: URL: https://github.com/apache/doris/pull/33700#issuecomment-2065903143 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compaction)Fix single compaction to get all local versions [doris]
yiguolei merged PR #33849: URL: https://github.com/apache/doris/pull/33849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](compaction)Fix single compaction to get all local versions #33849
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new d839908214a [fix](compaction)Fix single compaction to get all local
versions #33849
d839908214a is described below
commit d839908214a5139001469f02f6b173f6f1e1e9a3
Author: Sun Chenyang
AuthorDate: Fri Apr 19 15:14:36 2024 +0800
[fix](compaction)Fix single compaction to get all local versions #33849
add test and comment
---
be/src/olap/olap_server.cpp | 2 +-
be/src/olap/single_replica_compaction.cpp | 6 ++
be/src/olap/tablet.cpp| 11 +-
be/src/olap/tablet.h | 4 +++-
be/test/olap/tablet_test.cpp | 36 +++
5 files changed, 48 insertions(+), 11 deletions(-)
diff --git a/be/src/olap/olap_server.cpp b/be/src/olap/olap_server.cpp
index a2b5bf56de9..ce805b842dd 100644
--- a/be/src/olap/olap_server.cpp
+++ b/be/src/olap/olap_server.cpp
@@ -800,7 +800,7 @@ void StorageEngine::get_tablet_rowset_versions(const
PGetTabletVersionsRequest*
response->mutable_status()->set_status_code(TStatusCode::CANCELLED);
return;
}
-std::vector local_versions = tablet->get_all_versions();
+std::vector local_versions = tablet->get_all_local_versions();
for (const auto& local_version : local_versions) {
auto version = response->add_versions();
version->set_first(local_version.first);
diff --git a/be/src/olap/single_replica_compaction.cpp
b/be/src/olap/single_replica_compaction.cpp
index 038c3893497..9f0183a942c 100644
--- a/be/src/olap/single_replica_compaction.cpp
+++ b/be/src/olap/single_replica_compaction.cpp
@@ -217,10 +217,8 @@ Status
SingleReplicaCompaction::_get_rowset_verisons_from_peer(
bool SingleReplicaCompaction::_find_rowset_to_fetch(const
std::vector& peer_versions,
Version* proper_version) {
-// get local versions
-std::vector local_versions = _tablet->get_all_versions();
-std::sort(local_versions.begin(), local_versions.end(),
- [](const Version& left, const Version& right) { return
left.first < right.first; });
+// already sorted
+std::vector local_versions = tablet()->get_all_local_versions();
for (const auto& v : local_versions) {
VLOG_CRITICAL << _tablet->tablet_id() << " tablet local version: " <<
v.first << " - "
<< v.second;
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index e6f421d5ebe..22dac0a6312 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -1895,13 +1895,14 @@ void
Tablet::execute_single_replica_compaction(SingleReplicaCompaction& compacti
set_last_failure_time(this, compaction, 0);
}
-std::vector Tablet::get_all_versions() {
+std::vector Tablet::get_all_local_versions() {
std::vector local_versions;
{
-std::lock_guard wrlock(_meta_lock);
-SCOPED_SIMPLE_TRACE_IF_TIMEOUT(TRACE_TABLET_LOCK_THRESHOLD);
-for (const auto& it : _rs_version_map) {
-local_versions.emplace_back(it.first);
+std::shared_lock rlock(_meta_lock);
+for (const auto& [version, rs] : _rs_version_map) {
+if (rs->is_local()) {
+local_versions.emplace_back(version);
+}
}
}
std::sort(local_versions.begin(), local_versions.end(),
diff --git a/be/src/olap/tablet.h b/be/src/olap/tablet.h
index 8b28b57897d..95c70196354 100644
--- a/be/src/olap/tablet.h
+++ b/be/src/olap/tablet.h
@@ -287,7 +287,9 @@ public:
const std::set& alter_index_uids, bool is_drop_op);
std::vector
pick_candidate_rowsets_to_single_replica_compaction();
-std::vector get_all_versions();
+// used for single compaction to get the local versions
+// Single compaction does not require remote rowsets and cannot violate
the cooldown semantics
+std::vector get_all_local_versions();
std::vector pick_first_consecutive_empty_rowsets(int
limit);
diff --git a/be/test/olap/tablet_test.cpp b/be/test/olap/tablet_test.cpp
index f5778d47982..8d84b5141c3 100644
--- a/be/test/olap/tablet_test.cpp
+++ b/be/test/olap/tablet_test.cpp
@@ -148,6 +148,20 @@ public:
pb1->set_tablet_schema(_tablet_meta->tablet_schema());
}
+void init_rs_meta_resource(RowsetMetaSharedPtr& pb1, int64_t start,
int64_t end,
+ bool is_local) {
+RowsetMetaPB rowset_meta_pb;
+json2pb::JsonToProtoMessage(_json_rowset_meta, &rowset_meta_pb);
+rowset_meta_pb.set_start_version(start);
+rowset_meta_pb.set_end_version(end);
+rowset_meta_pb.set_creation_time(1);
+if (!is_local) {
+rowset_meta_pb.set_resource_id("100");
+}
+pb1->ini
Re: [PR] [fix](schema change) fix schema change check does not calculate reader merged rows [doris]
yiguolei merged PR #33825: URL: https://github.com/apache/doris/pull/33825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](backend) Add cpu core count and memory usage to "show backends" command. [doris]
github-actions[bot] commented on PR #33885: URL: https://github.com/apache/doris/pull/33885#issuecomment-2065907897 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](Nereids) Enable parse sql from sql cache and fix some bugs [doris]
yiguolei merged PR #33867: URL: https://github.com/apache/doris/pull/33867 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (13c110863b5 -> 411d146468c)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 13c110863b5 [exec](table_fun) opt bitmap/split vexplode table func performance (#33876) add 411d146468c [fix](schema change) fix schema change check does not calculate reader merged rows (#33825) No new revisions were added by this update. Summary of changes: be/src/common/config.cpp | 2 ++ be/src/common/config.h| 2 ++ be/src/olap/rowset/beta_rowset_reader.cpp | 3 +++ be/src/olap/rowset/beta_rowset_reader.h | 3 +++ be/src/olap/rowset/rowset_reader.h| 2 ++ be/src/olap/schema_change.h | 12 +--- 6 files changed, 21 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](DECIMAL) error DECIMAL cat to BOOLEAN [doris]
github-actions[bot] commented on PR #33868: URL: https://github.com/apache/doris/pull/33868#issuecomment-2065926474 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bug](fold) fix fold date/datetime error as null [doris]
zhangstar333 commented on PR #33845: URL: https://github.com/apache/doris/pull/33845#issuecomment-2065917047 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](DECIMAL) error DECIMAL cat to BOOLEAN [doris]
Mryange commented on PR #33868: URL: https://github.com/apache/doris/pull/33868#issuecomment-2065927748 > https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html#function_cast > > cast function of mysql do not support using bool as type argument. But there is implicit conversion to boolean. https://dbfiddle.uk/FGUsvxTk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (e66ca8c5d3f -> 0ac7849a9d3)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git discard e66ca8c5d3f [enhancement](Nereids) Enable parse sql from sql cache and fix some bugs (#33867) omit d839908214a [fix](compaction)Fix single compaction to get all local versions #33849 add 15f8014e4ea [enhancement](Nereids) Enable parse sql from sql cache and fix some bugs (#33867) add 0ac7849a9d3 [exec](table_fun) opt bitmap/split vexplode table func performance (#33876) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (e66ca8c5d3f) \ N -- N -- N refs/heads/branch-2.1 (0ac7849a9d3) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: be/src/olap/olap_server.cpp| 2 +- be/src/olap/single_replica_compaction.cpp | 6 ++-- be/src/olap/tablet.cpp | 11 +++ be/src/olap/tablet.h | 4 +-- .../vec/exprs/table_function/vexplode_bitmap.cpp | 28 + be/src/vec/exprs/table_function/vexplode_bitmap.h | 2 ++ be/src/vec/exprs/table_function/vexplode_split.cpp | 33 +--- be/src/vec/exprs/table_function/vexplode_split.h | 3 +- be/test/olap/tablet_test.cpp | 36 -- 9 files changed, 71 insertions(+), 54 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](DECIMAL) error DECIMAL cat to BOOLEAN [doris]
Mryange commented on PR #33868: URL: https://github.com/apache/doris/pull/33868#issuecomment-2065927964 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](node) Remove the cse DCHECK from the constructor. [doris]
github-actions[bot] commented on PR #33856: URL: https://github.com/apache/doris/pull/33856#issuecomment-2065939447 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](node) Remove the cse DCHECK from the constructor. [doris]
github-actions[bot] commented on PR #33856: URL: https://github.com/apache/doris/pull/33856#issuecomment-2065939547 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](runtime-filter) adjust need_local_merge setting conditions [doris]
doris-robot commented on PR #33886: URL: https://github.com/apache/doris/pull/33886#issuecomment-2065944111 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](variable) modify @@auto_commit column type to BIGINT in Nereids. [doris]
doris-robot commented on PR #33887: URL: https://github.com/apache/doris/pull/33887#issuecomment-2065944960 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](variable) modify @@auto_commit column type to BIGINT in Nereids. [doris]
morningman opened a new pull request, #33887: URL: https://github.com/apache/doris/pull/33887 ## Proposed changes #33282 only change the type in old planner. This PR modify it in Nereids ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compaction) fix single compaction input rowsets [doris]
github-actions[bot] commented on code in PR #33851:
URL: https://github.com/apache/doris/pull/33851#discussion_r1571954273
##
be/test/olap/tablet_test.cpp:
##
@@ -148,6 +148,20 @@ class TestTablet : public testing::Test {
pb1->set_tablet_schema(_tablet_meta->tablet_schema());
}
+void init_rs_meta_resource(RowsetMetaSharedPtr& pb1, int64_t start,
int64_t end,
Review Comment:
warning: method 'init_rs_meta_resource' can be made static
[readability-convert-member-functions-to-static]
```suggestion
static void init_rs_meta_resource(RowsetMetaSharedPtr& pb1, int64_t
start, int64_t end,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](node) Remove the cse DCHECK from the constructor. [doris]
BiteThet merged PR #33856: URL: https://github.com/apache/doris/pull/33856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [refine](node) Remove the cse DCHECK from the constructor (#33856)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0da148ec494 [refine](node) Remove the cse DCHECK from the constructor
(#33856)
0da148ec494 is described below
commit 0da148ec4946db6ba4250541ce9a8e14e7764182
Author: Mryange <59914473+mrya...@users.noreply.github.com>
AuthorDate: Fri Apr 19 15:43:17 2024 +0800
[refine](node) Remove the cse DCHECK from the constructor (#33856)
It's possible that a failure in the fe caused the check to fail, and at
that moment, it may not be possible to retrieve the corresponding query ID from
be.out.
---
be/src/exec/exec_node.cpp | 16 +---
be/src/pipeline/pipeline_x/operator.cpp | 13 +
be/src/pipeline/pipeline_x/operator.h | 3 ---
3 files changed, 26 insertions(+), 6 deletions(-)
diff --git a/be/src/exec/exec_node.cpp b/be/src/exec/exec_node.cpp
index eebb93cf1e7..382deaff6f4 100644
--- a/be/src/exec/exec_node.cpp
+++ b/be/src/exec/exec_node.cpp
@@ -90,10 +90,7 @@ ExecNode::ExecNode(ObjectPool* pool, const TPlanNode& tnode,
const DescriptorTbl
descs, std::vector {tnode.output_tuple_id}, std::vector
{true});
}
if (!tnode.intermediate_output_tuple_id_list.empty()) {
-DCHECK(tnode.__isset.output_tuple_id) << " no final output tuple id";
// common subexpression elimination
-DCHECK_EQ(tnode.intermediate_output_tuple_id_list.size(),
- tnode.intermediate_projections_list.size());
_intermediate_output_row_descriptor.reserve(tnode.intermediate_output_tuple_id_list.size());
for (auto output_tuple_id : tnode.intermediate_output_tuple_id_list) {
_intermediate_output_row_descriptor.push_back(
@@ -108,6 +105,19 @@ ExecNode::~ExecNode() = default;
Status ExecNode::init(const TPlanNode& tnode, RuntimeState* state) {
init_runtime_profile(get_name());
+if (!tnode.intermediate_output_tuple_id_list.empty()) {
+if (!tnode.__isset.output_tuple_id) {
+return Status::InternalError("no final output tuple id");
+}
+if (tnode.intermediate_output_tuple_id_list.size() !=
+tnode.intermediate_projections_list.size()) {
+return Status::InternalError(
+"intermediate_output_tuple_id_list size:{} not match "
+"intermediate_projections_list size:{}",
+tnode.intermediate_output_tuple_id_list.size(),
+tnode.intermediate_projections_list.size());
+}
+}
if (tnode.__isset.vconjunct) {
vectorized::VExprContextSPtr context;
diff --git a/be/src/pipeline/pipeline_x/operator.cpp
b/be/src/pipeline/pipeline_x/operator.cpp
index fab80e6ccbd..d5afad15fa0 100644
--- a/be/src/pipeline/pipeline_x/operator.cpp
+++ b/be/src/pipeline/pipeline_x/operator.cpp
@@ -110,6 +110,19 @@ std::string OperatorXBase::debug_string(RuntimeState*
state, int indentation_lev
Status OperatorXBase::init(const TPlanNode& tnode, RuntimeState* /*state*/) {
std::string node_name = print_plan_node_type(tnode.node_type);
+if (!tnode.intermediate_output_tuple_id_list.empty()) {
+if (!tnode.__isset.output_tuple_id) {
+return Status::InternalError("no final output tuple id");
+}
+if (tnode.intermediate_output_tuple_id_list.size() !=
+tnode.intermediate_projections_list.size()) {
+return Status::InternalError(
+"intermediate_output_tuple_id_list size:{} not match "
+"intermediate_projections_list size:{}",
+tnode.intermediate_output_tuple_id_list.size(),
+tnode.intermediate_projections_list.size());
+}
+}
auto substr = node_name.substr(0, node_name.find("_NODE"));
_op_name = substr + "_OPERATOR";
diff --git a/be/src/pipeline/pipeline_x/operator.h
b/be/src/pipeline/pipeline_x/operator.h
index b470c9237e8..efa35b2c2fe 100644
--- a/be/src/pipeline/pipeline_x/operator.h
+++ b/be/src/pipeline/pipeline_x/operator.h
@@ -164,10 +164,7 @@ public:
descs, std::vector {tnode.output_tuple_id}, std::vector
{true});
}
if (!tnode.intermediate_output_tuple_id_list.empty()) {
-DCHECK(tnode.__isset.output_tuple_id) << " no final output tuple
id";
// common subexpression elimination
-DCHECK_EQ(tnode.intermediate_output_tuple_id_list.size(),
- tnode.intermediate_projections_list.size());
_intermediate_output_row_descriptor.reserve(
tnode.intermediate_output_tuple_id_list.size());
for (auto output_tuple_id :
tnode.intermediate_output_tuple_id_list) {
-
Re: [PR] [enhancement](threadname) change thread name, whose length is restric… [doris]
caizj commented on PR #33792: URL: https://github.com/apache/doris/pull/33792#issuecomment-2065989258 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature] (partition) Dynamic partition behavior changes [doris]
github-actions[bot] commented on PR #33712: URL: https://github.com/apache/doris/pull/33712#issuecomment-2065990306 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Fix](compile) Fix gcc compile on master (#33864)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new dd470f30d65 [Fix](compile) Fix gcc compile on master (#33864) dd470f30d65 is described below commit dd470f30d65fced625e3ab06f424d102d6435cf0 Author: zclllyybb AuthorDate: Fri Apr 19 15:50:11 2024 +0800 [Fix](compile) Fix gcc compile on master (#33864) This is imported by #33511. wrongly used ColumnStr (); which violate C++20 standard(see https://wg21.cmeerw.net/cwg/issue2237) but still supported by clang up until now(see llvm/llvm-project#58112) --- be/src/exec/rowid_fetcher.h| 8 +--- be/src/exprs/hybrid_set.h | 4 be/src/olap/comparison_predicate.h | 3 +++ be/src/olap/in_list_predicate.h| 2 ++ be/src/olap/like_column_predicate.h| 2 ++ be/src/olap/match_predicate.h | 1 + be/src/olap/page_cache.h | 1 + .../rowset/segment_v2/hierarchical_data_reader.cpp | 2 ++ be/src/pipeline/exec/exchange_sink_buffer.cpp | 1 + be/src/pipeline/exec/exchange_sink_buffer.h| 1 + be/src/pipeline/exec/set_probe_sink_operator.cpp | 2 ++ be/src/pipeline/exec/set_sink_operator.cpp | 1 + be/src/pipeline/exec/set_source_operator.cpp | 1 + be/src/runtime/jsonb_value.h | 11 ++ be/src/runtime/primitive_type.h| 10 ++--- be/src/runtime/snapshot_loader.cpp | 1 + be/src/util/easy_json.cc | 1 + be/src/util/timezone_utils.cpp | 1 + .../aggregate_function_orthogonal_bitmap.h | 16 ++- .../aggregate_function_percentile_approx.h | 1 + .../aggregate_function_reader_first_last.h | 4 .../aggregate_function_window.h| 4 be/src/vec/columns/column.h| 4 be/src/vec/columns/column_array.cpp| 4 be/src/vec/columns/column_complex.h| 3 +++ be/src/vec/columns/column_const.h | 1 + be/src/vec/columns/column_decimal.cpp | 1 + be/src/vec/columns/column_dictionary.h | 12 +++ be/src/vec/columns/column_dummy.h | 5 + be/src/vec/columns/column_fixed_length_object.h| 6 ++ be/src/vec/columns/column_map.cpp | 1 + be/src/vec/columns/column_map.h| 1 + be/src/vec/columns/column_nullable.cpp | 2 ++ be/src/vec/columns/column_nullable.h | 1 + be/src/vec/columns/column_object.h | 3 +++ be/src/vec/columns/column_string.cpp | 1 + be/src/vec/columns/column_string.h | 16 ++- be/src/vec/columns/column_struct.cpp | 4 be/src/vec/columns/column_struct.h | 1 + be/src/vec/columns/column_vector.cpp | 1 + be/src/vec/columns/column_vector.h | 1 + be/src/vec/columns/columns_common.h| 1 + be/src/vec/columns/predicate_column.h | 15 ++ be/src/vec/common/field_visitors.h | 1 + be/src/vec/core/decimal_comparison.h | 6 ++ be/src/vec/core/field.h| 11 ++ be/src/vec/data_types/data_type_decimal.h | 1 + be/src/vec/data_types/data_type_nothing.cpp| 2 ++ be/src/vec/data_types/data_type_struct.cpp | 3 +++ be/src/vec/data_types/data_type_struct.h | 1 + .../vec/data_types/serde/data_type_number_serde.h | 1 + .../exec/format/parquet/byte_array_dict_decoder.h | 16 +-- be/src/vec/exec/format/parquet/decoder.h | 24 -- .../exec/format/parquet/parquet_column_convert.h | 2 +- .../format/parquet/vparquet_column_chunk_reader.h | 13 +++- .../exec/format/parquet/vparquet_column_reader.cpp | 9 .../exec/format/parquet/vparquet_column_reader.h | 17 +++ be/src/vec/exec/format/table/iceberg_reader.h | 11 +- be/src/vec/exec/vdata_gen_scan_node.cpp| 1 + be/src/vec/exec/vset_operation_node.cpp| 4 be/src/vec/exprs/table_function/vexplode_split.h | 17 +++ .../array/function_array_enumerate_uniq.cpp| 1 + .../vec/functions/array/function_arrays_overlap.h | 10 - be/src/vec/functions/function.h| 1 + be/src/vec/functions/function_cast.h | 2 ++ be/src/vec/functions/function_jsonb.cpp| 1 + be/src/vec/functions/function_variadic_arguments.h | 1 + be/src/vec/functions/in.h | 23 -
Re: [PR] [Fix](compile) Fix gcc compile on master [doris]
BiteThet merged PR #33864: URL: https://github.com/apache/doris/pull/33864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
BiteThet commented on PR #33496: URL: https://github.com/apache/doris/pull/33496#issuecomment-2066001331 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [func](jsonb)support json_depth function [doris]
doris-robot commented on PR #24801: URL: https://github.com/apache/doris/pull/24801#issuecomment-2066008811 TeamCity be ut coverage result: Function Coverage: 35.42% (8923/25192) Line Coverage: 27.14% (73314/270086) Region Coverage: 26.25% (37844/144176) Branch Coverage: 23.07% (19289/83628) Coverage Report: http://coverage.selectdb-in.cc/coverage/62e5b11d7a37f93de8b5e19268011cb47695fe1f_62e5b11d7a37f93de8b5e19268011cb47695fe1f/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](coord) use single instance only with small limit [doris]
morningman opened a new pull request, #33888: URL: https://github.com/apache/doris/pull/33888 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](coord) use single instance only with small limit [doris]
doris-robot commented on PR #33888: URL: https://github.com/apache/doris/pull/33888#issuecomment-2066014040 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [docker](k8s)k8s docker script support create account and remove some useless code [doris]
LemonLiTree opened a new pull request, #33889: URL: https://github.com/apache/doris/pull/33889 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [docker](k8s)k8s docker script support create account and remove some useless code [doris]
doris-robot commented on PR #33889: URL: https://github.com/apache/doris/pull/33889#issuecomment-2066018718 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [docker](k8s)k8s docker script support create account and remove some useless code [doris]
LemonLiTree commented on PR #33889: URL: https://github.com/apache/doris/pull/33889#issuecomment-2066026135 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](coord) use single instance only with small limit [doris]
morningman commented on PR #33888: URL: https://github.com/apache/doris/pull/33888#issuecomment-2066039524 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](coord) use single instance only with small limit [doris]
github-actions[bot] commented on PR #33888: URL: https://github.com/apache/doris/pull/33888#issuecomment-2066044577 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](coord) use single instance only with small limit [doris]
github-actions[bot] commented on PR #33888: URL: https://github.com/apache/doris/pull/33888#issuecomment-2066044670 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] add document for powerbi [doris-website]
wyx123654 opened a new pull request, #576: URL: https://github.com/apache/doris-website/pull/576 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Performance](Variant) Improve load performance for variant type [doris]
doris-robot commented on PR #33890: URL: https://github.com/apache/doris/pull/33890#issuecomment-2066069695 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Performance](Variant) Improve load performance for variant type [doris]
eldenmoon commented on PR #33890: URL: https://github.com/apache/doris/pull/33890#issuecomment-2066070135 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] 使用paimon catalog 查询带有is null报错 [doris]
Muzi999 commented on issue #30037: URL: https://github.com/apache/doris/issues/30037#issuecomment-2066071904 > ### Search before asking > * [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.我在问题中搜索过，没有发现类似的问题。 > > ### Version > 2.0.3 > > ### What's Wrong? > 1. 挂载paimon catalog > > CREATE CATALOG `paimon_s3` PROPERTIES (创建目录 `paimon_s3` 属性 ( "type" = "paimon", “类型”=“派蒙”， "warehouse" = "s3://paimon-1308700295.cos.ap-beijing.myqcloud.com/paimoncos","仓库" = "s3://paimon-1308700295.cos.ap-beijing.myqcloud.com/paimoncos", "s3.endpoint" = "cos.ap-beijing.myqcloud.com", "s3.access_key" = "ak", “s3.access_key”=“ak”， "s3.secret_key" = "sk" “s3.secret_key”=“sk” ); > > 2. 执行sql >switch paimon_s3; >use `default`; >select * from order where orderId is null; > > 3.报错信息 > > 2024-01-17 09:39:48,842 WARN (mysql-nio-pool-132|105023) [StmtExecutor.analyze():1009] Analyze failed. stmt[2804, f7527339f184618-8e4cea723f63acea]2024-01-17 09:39:48,842 警告 (mysql-nio-pool-132|105023) [StmtExecutor.analyze():1009] 分析失败。 stmt[2804, f7527339f184618-8e4cea723f63acea] java.lang.ClassCastException: org.apache.doris.analysis.IsNullPredicate cannot be cast to org.apache.doris.analysis.BinaryPredicatejava.lang.ClassCastException：org.apache.doris.analysis.IsNullPredicate 无法转换为 org.apache.doris.analysis.BinaryPredicate at org.apache.doris.planner.external.paimon.PaimonPredicateConverter.binaryExprDesc(PaimonPredicateConverter.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.external.paimon.PaimonPredicateConverter.binaryExprDesc(PaimonPredicateConverter.java:87) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.external.paimon.PaimonPredicateConverter.convertToPaimonExpr(PaimonPredicateConverter.java:81) ~[doris-fe.jar:1. 2-SNAPSHOT]在 org.apache.doris.planner.external.paimon.PaimonPredicateConverter.convertToPaimonExpr(PaimonPredicateConverter.java:81) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.external.paimon.PaimonPredicateConverter.convertToPaimonExpr(PaimonPredicateConverter.java:53) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.external.paimon.PaimonPredicateConverter.convertToPaimonExpr(PaimonPredicateConverter.java:53) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.external.paimon.PaimonScanNode.doInitialize(PaimonScanNode.java:79) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.external.paimon.PaimonScanNode.doInitialize(PaimonScanNode.java:79) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.external.FileQueryScanNode.init(FileQueryScanNode.java:119) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.external.FileQueryScanNode.init(FileQueryScanNode.java:119) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.SingleNodePlanner .createScanNode(SingleNodePlanner.java:2057) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:2057) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2206) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2206) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:1242) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:1242) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:264) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:264) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.SingleNodePlanner.createInlineViewPlan(SingleNodeP lanner.java:1663) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.SingleNodePlanner.createInlineViewPlan(SingleNodePlanner.java:1663) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2210) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2210) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:1242) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:1242) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:264) ~[doris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:264) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:187) ~[do ris-fe.jar:1.2-SNAPSHOT]在 org.apache.doris.planner.SingleNodePlanner.createSingl
Re: [PR] [feature](java-udtf) support java-udtf [doris]
doris-robot commented on PR #33595: URL: https://github.com/apache/doris/pull/33595#issuecomment-2066071964 TeamCity be ut coverage result: Function Coverage: 35.40% (8920/25200) Line Coverage: 27.12% (73282/270214) Region Coverage: 26.24% (37839/144228) Branch Coverage: 23.06% (19284/83630) Coverage Report: http://coverage.selectdb-in.cc/coverage/43496cce48500510628cb02188df8d34de5b3c02_43496cce48500510628cb02188df8d34de5b3c02/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] 使用paimon catalog 查询带有is null报错 [doris]
Muzi999 commented on issue #30037: URL: https://github.com/apache/doris/issues/30037#issuecomment-2066074107 I'm also experiencing this problem, what can I do to solve it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](mysql) support prepare statement completely [doris]
nextdreamblue commented on PR #33403: URL: https://github.com/apache/doris/pull/33403#issuecomment-2066079677 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](decompress)(review) context leaked in failure path [doris]
yiguolei merged PR #33622: URL: https://github.com/apache/doris/pull/33622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (dd470f30d65 -> aadc1a6acca)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from dd470f30d65 [Fix](compile) Fix gcc compile on master (#33864) add aadc1a6acca [fix](decompress)(review) context leaked in failure path (#33622) No new revisions were added by this update. Summary of changes: be/src/util/block_compression.cpp | 298 ++ 1 file changed, 143 insertions(+), 155 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix][mow] free _memtable_flush_executor and _calc_delete_bitmap_executor resource before stop CloudStorageEngine [doris]
hust-hhb opened a new pull request, #33891: URL: https://github.com/apache/doris/pull/33891 ## Proposed changes _memtable_flush_executor and _calc_delete_bitmap_executor should free before stop CloudStorageEngine,like StorageEngine does. Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix][mow] free _memtable_flush_executor and _calc_delete_bitmap_executor resource before stop CloudStorageEngine [doris]
doris-robot commented on PR #33891: URL: https://github.com/apache/doris/pull/33891#issuecomment-2066088065 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix][mow] free _memtable_flush_executor and _calc_delete_bitmap_executor resource before stop CloudStorageEngine [doris]
hust-hhb commented on PR #33891: URL: https://github.com/apache/doris/pull/33891#issuecomment-2066088434 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature] (partition) Dynamic partition behavior changes [doris]
HappenLee merged PR #33712: URL: https://github.com/apache/doris/pull/33712 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feature] (partition) Dynamic partition behavior changes (#33712)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 74ce738d367 [feature] (partition) Dynamic partition behavior changes
(#33712)
74ce738d367 is described below
commit 74ce738d3672004d2a453d3f6d17395db70321d2
Author: Uniqueyou <134280716+wyxxx...@users.noreply.github.com>
AuthorDate: Fri Apr 19 16:29:32 2024 +0800
[feature] (partition) Dynamic partition behavior changes (#33712)
---
.../doris/clone/DynamicPartitionScheduler.java | 7 +--
.../test_dynamic_partition.groovy | 73 ++
2 files changed, 74 insertions(+), 6 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
index f09c83da3fc..8ed3c8c14e2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
@@ -422,18 +422,13 @@ public class DynamicPartitionScheduler extends
MasterDaemon {
ZonedDateTime now =
ZonedDateTime.now(dynamicPartitionProperty.getTimeZone().toZoneId());
String lowerBorder =
DynamicPartitionUtil.getPartitionRangeString(dynamicPartitionProperty,
now, dynamicPartitionProperty.getStart(), partitionFormat);
-String upperBorder =
DynamicPartitionUtil.getPartitionRangeString(dynamicPartitionProperty,
-now, dynamicPartitionProperty.getEnd() + 1, partitionFormat);
PartitionValue lowerPartitionValue = new PartitionValue(lowerBorder);
-PartitionValue upperPartitionValue = new PartitionValue(upperBorder);
List> reservedHistoryPartitionKeyRangeList = new
ArrayList>();
Range reservePartitionKeyRange;
try {
PartitionKey lowerBound =
PartitionKey.createPartitionKey(Collections.singletonList(lowerPartitionValue),
Collections.singletonList(partitionColumn));
-PartitionKey upperBound =
PartitionKey.createPartitionKey(Collections.singletonList(upperPartitionValue),
-Collections.singletonList(partitionColumn));
-reservePartitionKeyRange = Range.closedOpen(lowerBound,
upperBound);
+reservePartitionKeyRange = Range.atLeast(lowerBound);
reservedHistoryPartitionKeyRangeList.add(reservePartitionKeyRange);
} catch (AnalysisException | IllegalArgumentException e) {
// AnalysisException: keys.size is always equal to column.size,
cannot reach this exception
diff --git
a/regression-test/suites/partition_p1/dynamic_partition/test_dynamic_partition.groovy
b/regression-test/suites/partition_p1/dynamic_partition/test_dynamic_partition.groovy
new file mode 100644
index 000..a0434bb0c10
--- /dev/null
+++
b/regression-test/suites/partition_p1/dynamic_partition/test_dynamic_partition.groovy
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_dynamic_partition_with_update","nonConcurrent") {
+def tbl = "test_dynamic_partition_with_update"
+sql "drop table if exists ${tbl}"
+sql """
+CREATE TABLE IF NOT EXISTS ${tbl}
+( k1 date NOT NULL )
+PARTITION BY RANGE(k1) ( )
+DISTRIBUTED BY HASH(k1) BUCKETS 1
+PROPERTIES (
+"dynamic_partition.enable"="true",
+"dynamic_partition.end"="3",
+"dynamic_partition.buckets"="1",
+"dynamic_partition.start"="-3",
+"dynamic_partition.prefix"="p",
+"dynamic_partition.time_unit"="DAY",
+"dynamic_partition.create_history_partition"="true",
+"dynamic_partition.replication_allocation" =
"tag.location.default: 1",
+"replication_num" = "1");
+"""
+
+// set check interval time
+sql """ admin set frontend config
('dynamic_partition_check_interval_seconds' = '2') """
+
+// check table init
+def result = sql "show partitions
Re: [PR] [feature](iceberg) add iceberg transaction implement [doris]
wuwenchi commented on PR #33629: URL: https://github.com/apache/doris/pull/33629#issuecomment-2066095032 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
wuwenchi commented on PR #33883: URL: https://github.com/apache/doris/pull/33883#issuecomment-2066095207 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Performance](Variant) Improve load performance for variant type [doris]
github-actions[bot] commented on code in PR #33890:
URL: https://github.com/apache/doris/pull/33890#discussion_r1572029446
##
be/src/vec/columns/column_object.cpp:
##
@@ -148,6 +151,84 @@ class FieldVisitorToNumberOfDimensions : public
StaticVisitor {
}
};
+// Visitor that allows to get type of scalar field
+// but include contain complex field.This is a faster version
+// for FieldVisitorToScalarType which does not support complex field.
+class SimpleFieldVisitorToScarlarType : public StaticVisitor {
+public:
+size_t operator()(const Array& x) {
+throw doris::Exception(ErrorCode::INVALID_ARGUMENT, "Array type is not
supported");
+}
+size_t operator()(const UInt64& x) {
+if (x <= std::numeric_limits::max()) {
+type = TypeIndex::Int8;
+} else if (x <= std::numeric_limits::max()) {
+type = TypeIndex::Int16;
+} else if (x <= std::numeric_limits::max()) {
+type = TypeIndex::Int32;
+} else {
+type = TypeIndex::Int64;
+}
+return 1;
+}
+size_t operator()(const Int64& x) {
+if (x <= std::numeric_limits::max() && x >=
std::numeric_limits::min()) {
+type = TypeIndex::Int8;
+} else if (x <= std::numeric_limits::max() &&
+ x >= std::numeric_limits::min()) {
+type = TypeIndex::Int16;
+} else if (x <= std::numeric_limits::max() &&
+ x >= std::numeric_limits::min()) {
+type = TypeIndex::Int32;
+} else {
+type = TypeIndex::Int64;
+}
+return 1;
+}
+size_t operator()(const JsonbField& x) {
+type = TypeIndex::JSONB;
+return 1;
+}
+size_t operator()(const Null&) {
+have_nulls = true;
+return 1;
+}
+template
+size_t operator()(const T&) {
+type = TypeId>::value;
+return 1;
+}
+void get_scalar_type(DataTypePtr* data_type) const {
+WhichDataType which(type);
+#define DISPATCH(TYPE) \
+if (which.idx == TypeIndex::TYPE) {\
+*data_type = std::make_shared>(); \
+return;\
+}
+FOR_NUMERIC_TYPES(DISPATCH)
+#undef DISPATCH
+if (which.is_string()) {
+*data_type = std::make_shared();
+return;
+}
+if (which.is_json()) {
+*data_type = std::make_shared();
+return;
+}
+if (which.is_nothing()) {
+*data_type = std::make_shared();
+return;
+}
+}
+bool contain_nulls() const { return have_nulls; }
+
+bool need_convert_field() const { return false; }
Review Comment:
warning: method 'need_convert_field' can be made static
[readability-convert-member-functions-to-static]
```suggestion
static bool need_convert_field() { return false; }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](nereids) fix rule merge_aggregate when has project [doris]
feiniaofeiafei opened a new pull request, #33892: URL: https://github.com/apache/doris/pull/33892 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](nereids) fix rule merge_aggregate when has project [doris]
doris-robot commented on PR #33892: URL: https://github.com/apache/doris/pull/33892#issuecomment-2066099648 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](backend) Add cpu core count and memory usage to "show backends" command. [doris]
doris-robot commented on PR #33885: URL: https://github.com/apache/doris/pull/33885#issuecomment-2066099770 TeamCity be ut coverage result: Function Coverage: 35.42% (8922/25188) Line Coverage: 27.15% (73323/270057) Region Coverage: 26.26% (37856/144146) Branch Coverage: 23.09% (19298/83590) Coverage Report: http://coverage.selectdb-in.cc/coverage/9cc2c3c3475df1bc7a9a50a3752fe1f42af59da7_9cc2c3c3475df1bc7a9a50a3752fe1f42af59da7/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](nereids) fix rule merge_aggregate when has project [doris]
feiniaofeiafei commented on PR #33892: URL: https://github.com/apache/doris/pull/33892#issuecomment-2066099789 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [PipelineX](improvement) Prepare tasks in parallel [doris]
Gabriel39 commented on PR #32789: URL: https://github.com/apache/doris/pull/32789#issuecomment-2066105062 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [chore](errmsg) Fix confusing error message and clang tidy hints [doris]
zclllyybb opened a new pull request, #33893: URL: https://github.com/apache/doris/pull/33893 ## Proposed changes Issue Number: close #xxx 1. removed clang-tidy options which nonsense or always false alarm: `readability-avoid-const-params-in-decls` and `readability-convert-member-functions-to-static` 2. for MoW table try use VALUE column as PARTITION column, before got: ``` The partition column could not be aggregated column ``` now: ``` Merge-on-Write table's partition column must be KEY column ``` ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](errmsg) Fix confusing error message and clang tidy hints [doris]
zclllyybb commented on PR #33893: URL: https://github.com/apache/doris/pull/33893#issuecomment-2066105222 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](backend) Add cpu core count and memory usage to "show backends" command. [doris]
Mryange commented on PR #33885: URL: https://github.com/apache/doris/pull/33885#issuecomment-2066103543 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](errmsg) Fix confusing error message and clang tidy hints [doris]
doris-robot commented on PR #33893: URL: https://github.com/apache/doris/pull/33893#issuecomment-2066105185 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](test)fix auto partition date file not exists [doris]
zclllyybb commented on PR #33827: URL: https://github.com/apache/doris/pull/33827#issuecomment-2066106544 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [PipelineX](improvement) Prepare tasks in parallel [doris]
github-actions[bot] commented on code in PR #32789:
URL: https://github.com/apache/doris/pull/32789#discussion_r1572037934
##
be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:
##
@@ -486,11 +490,17 @@
return Status::OK();
}
-Status PipelineXFragmentContext::_build_pipeline_tasks(
-const doris::TPipelineFragmentParams& request) {
+Status PipelineXFragmentContext::_build_pipeline_x_tasks(
Review Comment:
warning: function '_build_pipeline_x_tasks' exceeds recommended
size/complexity thresholds [readability-function-size]
```cpp
Status PipelineXFragmentContext::_build_pipeline_x_tasks(
^
```
Additional context
**be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:492:** 222
lines including whitespace and comments (threshold 80)
```cpp
Status PipelineXFragmentContext::_build_pipeline_x_tasks(
^
```
##
be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:
##
@@ -164,7 +168,8 @@ void PipelineXFragmentContext::cancel(const
PPlanFragmentCancelReason& reason,
}
}
-Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams&
request) {
+Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams&
request,
Review Comment:
warning: function 'prepare' exceeds recommended size/complexity thresholds
[readability-function-size]
```cpp
Status PipelineXFragmentContext::prepare(const
doris::TPipelineFragmentParams& request,
^
```
Additional context
**be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:170:** 105
lines including whitespace and comments (threshold 80)
```cpp
Status PipelineXFragmentContext::prepare(const
doris::TPipelineFragmentParams& request,
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](DECIMAL) error DECIMAL cat to BOOLEAN [doris]
zhiqiang- commented on PR #33868: URL: https://github.com/apache/doris/pull/33868#issuecomment-2066109596 > > https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html#function_cast > > cast function of mysql do not support using bool as type argument. > > But there is implicit conversion to boolean. https://dbfiddle.uk/FGUsvxTk why do you think there is an implicit conversion? https://dev.mysql.com/doc/refman/8.3/en/logical-operators.html#operator_and oprator_and checks whether operands are zero or NULL, it has nothing to do with type convert. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Revert "[improve](load) lock free flush_async and wait_flush in LoadChannel (#33389)" [doris]
xiaokang commented on PR #33871: URL: https://github.com/apache/doris/pull/33871#issuecomment-2066111849 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.0](routine-load) fix single stream multi table load cannot finish [doris]
xiaokang merged PR #33880: URL: https://github.com/apache/doris/pull/33880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [branch-2.0](routine-load) fix single stream multi table load cannot finish #33816 (#33880)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 42e69041c86 [branch-2.0](routine-load) fix single stream multi table
load cannot finish #33816 (#33880)
42e69041c86 is described below
commit 42e69041c866c87e11e87c9fdf5160e354aaff4f
Author: HHoflittlefish777 <77738092+hhoflittlefish...@users.noreply.github.com>
AuthorDate: Fri Apr 19 16:44:07 2024 +0800
[branch-2.0](routine-load) fix single stream multi table load cannot finish
#33816 (#33880)
---
be/src/common/config.cpp | 7 ++
be/src/common/config.h | 7 ++
be/src/io/fs/multi_table_pipe.cpp| 28 +++-
be/src/io/fs/multi_table_pipe.h | 3 +++
be/src/io/fs/stream_load_pipe.cpp| 9
be/src/io/fs/stream_load_pipe.h | 4
docs/en/docs/admin-manual/config/be-config.md| 6 +
docs/zh-CN/docs/admin-manual/config/be-config.md | 7 ++
8 files changed, 65 insertions(+), 6 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 2e715961348..8440433b707 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -790,6 +790,13 @@ DEFINE_Int32(routine_load_consumer_pool_size, "10");
// if the size of batch is more than this threshold, we will request plans for
all related tables.
DEFINE_Int32(multi_table_batch_plan_threshold, "200");
+// Used in single-stream-multi-table load. When receiving a batch of messages
from Kafka,
+// if the size of the table wait for plan is more than this threshold, we will
request plans for all related tables.
+// The param is aimed to avoid requesting and executing too many plans at once.
+// Performing small batch processing on multiple tables during the loaded
process can reduce the pressure of a single RPC
+// and improve the real-time processing of data.
+DEFINE_Int32(multi_table_max_wait_tables, "5");
+
// When the timeout of a load task is less than this threshold,
// Doris treats it as a high priority task.
// high priority tasks use a separate thread pool for flush and do not block
rpc by memory cleanup logic.
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 1947f791cb5..fcb501cb802 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -844,6 +844,13 @@ DECLARE_Int32(routine_load_consumer_pool_size);
// if the size of batch is more than this threshold, we will request plans for
all related tables.
DECLARE_Int32(multi_table_batch_plan_threshold);
+// Used in single-stream-multi-table load. When receiving a batch of messages
from Kafka,
+// if the size of the table wait for plan is more than this threshold, we will
request plans for all related tables.
+// The param is aimed to avoid requesting and executing too many plans at once.
+// Performing small batch processing on multiple tables during the loaded
process can reduce the pressure of a single RPC
+// and improve the real-time processing of data.
+DECLARE_Int32(multi_table_max_wait_tables);
+
// When the timeout of a load task is less than this threshold,
// Doris treats it as a high priority task.
// high priority tasks use a separate thread pool for flush and do not block
rpc by memory cleanup logic.
diff --git a/be/src/io/fs/multi_table_pipe.cpp
b/be/src/io/fs/multi_table_pipe.cpp
index 399676657f1..97f88161bc9 100644
--- a/be/src/io/fs/multi_table_pipe.cpp
+++ b/be/src/io/fs/multi_table_pipe.cpp
@@ -114,21 +114,37 @@ Status MultiTablePipe::dispatch(const std::string& table,
const char* data, size
} else {
pipe = iter->second;
}
-RETURN_NOT_OK_STATUS_WITH_WARN((pipe.get()->*cb)(data, size),
- "append failed in unplanned kafka
pipe");
+// It is necessary to determine whether the sum of
pipe_current_capacity and size is greater than pipe_max_capacity,
+// otherwise the following situation may occur:
+// the pipe is full but still cannot trigger the request and exec plan
condition,
+// causing one stream multi table load can not finish
++_unplanned_row_cnt;
-size_t threshold = config::multi_table_batch_plan_threshold;
-if (_unplanned_row_cnt >= threshold) {
-LOG(INFO) << fmt::format("unplanned row cnt={} reach threshold={},
plan them",
- _unplanned_row_cnt, threshold);
+auto pipe_current_capacity = pipe->current_capacity();
+auto pipe_max_capacity = pipe->max_capacity();
+if (_unplanned_row_cnt >= _row_threshold ||
+_unplanned_pipes.size() >= _wait_tables_threshold ||
+pipe_current_capacity + size > pipe_max_capacity) {
+LOG(INFO) << fmt::format(
+
Re: [PR] [fix][mow] free _memtable_flush_executor and _calc_delete_bitmap_executor resource before stop CloudStorageEngine [doris]
github-actions[bot] commented on PR #33891: URL: https://github.com/apache/doris/pull/33891#issuecomment-2066120537 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](mysql) support prepare statement completely [doris]
github-actions[bot] commented on PR #33403: URL: https://github.com/apache/doris/pull/33403#issuecomment-2066122163 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](MultiPartition)let the user set the prefix of the MultiParti… [doris]
github-actions[bot] commented on PR #33820: URL: https://github.com/apache/doris/pull/33820#issuecomment-2066126328 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](backend) Add cpu core count and memory usage to "show backends" command. [doris]
github-actions[bot] commented on PR #33885: URL: https://github.com/apache/doris/pull/33885#issuecomment-2066128085 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](MultiPartition)let the user set the prefix of the MultiParti… [doris]
github-actions[bot] commented on PR #33820: URL: https://github.com/apache/doris/pull/33820#issuecomment-2066126272 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](errmsg) Fix confusing error message and clang tidy hints [doris]
github-actions[bot] commented on PR #33893: URL: https://github.com/apache/doris/pull/33893#issuecomment-2066132360 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](auto-partition) Forbid use Auto and Dynamic partition at the same time [doris]
zclllyybb commented on PR #33736: URL: https://github.com/apache/doris/pull/33736#issuecomment-2066130826 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](DECIMAL) error DECIMAL cat to BOOLEAN [doris]
Mryange commented on PR #33868: URL: https://github.com/apache/doris/pull/33868#issuecomment-2066131162 > > > https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html#function_cast > > > cast function of mysql do not support using bool as type argument. > > > > > > But there is implicit conversion to boolean. https://dbfiddle.uk/FGUsvxTk > > why do you think there is an implicit conversion? > > https://dev.mysql.com/doc/refman/8.3/en/logical-operators.html#operator_and > > oprator_and checks whether operands are zero or NULL, it has nothing to do with type convert. My mistake, as doris implementation involves implicit conversion. But such results in doris ``` mysql []>set enable_fold_constant_by_be = false; Query OK, 0 rows affected (0.00 sec) mysql []>select 3.14 and 3.0; +--+ | (cast(3.14 as BOOLEAN) AND cast(3.0 as BOOLEAN)) | +--+ |3 | +--+ 1 row in set (0.03 sec) mysql []>desc select 3.14 and 3.0; +--+ | Explain String(Nereids Planner) | +--+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | (cast(3.14 as BOOLEAN) AND cast(3.0 as BOOLEAN))[#0] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 0:VUNION(30) | | constant exprs: | | CAST(3.14 AS BOOLEAN) AND CAST(3.0 AS BOOLEAN) | +--+ 13 rows in set (0.01 sec) mysql []>select 3.14 and 2.0; +--+ | (cast(3.14 as BOOLEAN) AND cast(2.0 as BOOLEAN)) | +--+ |2 | +--+ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](meta) Add serialize name for auto partition FE properties [doris]
zclllyybb opened a new pull request, #33894: URL: https://github.com/apache/doris/pull/33894 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
