(doris-website) branch master updated: [docs](set) enable admin_set_frontend_config can apply to all fe (#636)
This is an automated email from the ASF dual-hosted git repository.
lide pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 78899390b4 [docs](set) enable admin_set_frontend_config can apply to
all fe (#636)
78899390b4 is described below
commit 78899390b4f6a66fd6aa1258bb9ec18b89ddb967
Author: Yulei-Yang
AuthorDate: Mon May 13 13:58:26 2024 +0800
[docs](set) enable admin_set_frontend_config can apply to all fe (#636)
---
.../Database-Administration-Statements/ADMIN-SET-CONFIG.md| 7 ++-
.../Database-Administration-Statements/ADMIN-SET-CONFIG.md| 8 ++--
.../Database-Administration-Statements/ADMIN-SET-CONFIG.md| 7 ++-
.../Database-Administration-Statements/ADMIN-SET-CONFIG.md| 7 ++-
.../Database-Administration-Statements/ADMIN-SET-CONFIG.md| 7 ++-
.../Database-Administration-Statements/ADMIN-SET-CONFIG.md| 7 ++-
6 files changed, 36 insertions(+), 7 deletions(-)
diff --git
a/docs/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
b/docs/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
index 0d76164bdd..f59828f40d 100644
---
a/docs/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
+++
b/docs/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
@@ -38,9 +38,14 @@ The settable configuration items can be viewed through the
SHOW FRONTEND CONFIG;
grammar:
```sql
- ADMIN SET FRONTEND CONFIG ("key" = "value");
+ ADMIN SET FRONTEND CONFIG ("key" = "value") [ALL];
+ ADMIN SET ALL FRONTENDS CONFIG ("key" = "value");
+illustrate:
+
+1. If `ALL` keyword is used, this config will be applied to all FE(except
master_only configuration)
+
### Example
1. Set 'disable_balance' to true
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
index 7e8cd41ef9..0663934e77 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
@@ -38,9 +38,13 @@ ADMIN SET CONFIG
语法：
```sql
- ADMIN SET FRONTEND CONFIG ("key" = "value");
-```
+ ADMIN SET FRONTEND CONFIG ("key" = "value") [ALL];
+ ADMIN SET ALL FRONTENDS CONFIG ("key" = "value");```
+说明：
+
+1. 使用ALL关键字后配置参数将应用于所有FE(除 master_only 参数外)
+
### Example
1. 设置 'disable_balance' 为 true
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-SET-CONFIG.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-SET-CONFIG.md
index ca012b3b4c..3e6638578a 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-SET-CONFIG.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-SET-CONFIG.md
@@ -38,9 +38,14 @@ ADMIN SET CONFIG
语法：
```sql
- ADMIN SET FRONTEND CONFIG ("key" = "value");
+ ADMIN SET FRONTEND CONFIG ("key" = "value") [ALL];
+ ADMIN SET ALL FRONTENDS CONFIG ("key" = "value");
```
+说明：
+
+1. 使用ALL关键字后配置参数将应用于所有FE(除 master_only 参数外)
+
### Example
1. 设置 'disable_balance' 为 true
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
index 7e8cd41ef9..1b4feac596 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/Database-Administration-Statements/ADMIN-SET-CONFIG.md
@@ -38,9 +38,14 @@ ADMIN SET CONFIG
语法：
```sql
- ADMIN SET FRONTEND CONFIG ("key" = "value");
+ ADMIN SET FRONTEND CONFIG ("key" = "value") [ALL];
+ ADMIN SET ALL FRONTENDS CONFIG ("key" = "value");
```
+说明：
+
+1. 使用ALL关键字后配置参数将应用于所有FE(除 master_only 参数外)
+
### Example
1. 设置 'disable_balance' 为 true
diff --git
a/versioned_docs/version-2.0/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-SET-CONFIG.md
b/versioned_docs/version-2.0/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-SET-CONFIG.md
index 0b4728f965..db14df6e4a
Error while running notifications feature from .asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [docs](set) enable admin_set_frontend_config can apply to all fe [doris-website]
lide-reed merged PR #636: URL: https://github.com/apache/doris-website/pull/636 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](status) shoud use add error message method not using index [doris]
github-actions[bot] commented on PR #34754: URL: https://github.com/apache/doris/pull/34754#issuecomment-2106712776 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](status) shoud use add error message method not using index [doris]
github-actions[bot] commented on PR #34754: URL: https://github.com/apache/doris/pull/34754#issuecomment-2106712736 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) Avoid unnecessary copies in the profile thrift. [doris]
doris-robot commented on PR #34720: URL: https://github.com/apache/doris/pull/34720#issuecomment-2106712461 TeamCity be ut coverage result: Function Coverage: 35.65% (8986/25203) Line Coverage: 27.32% (74280/271889) Region Coverage: 26.55% (38386/144589) Branch Coverage: 23.37% (19576/83774) Coverage Report: http://coverage.selectdb-in.cc/coverage/de6bdf54940e3437c5a0ce1aba98112b24ca1ef8_de6bdf54940e3437c5a0ce1aba98112b24ca1ef8/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feat](Nereids): Reject Commutativity Swap for Nested Loop Joins Affecting Parallelism (#34639)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 141c3330b4e [feat](Nereids): Reject Commutativity Swap for Nested Loop
Joins Affecting Parallelism (#34639)
141c3330b4e is described below
commit 141c3330b4ebf5aaff874e72c711a6b2f6efddf5
Author: 谢健
AuthorDate: Mon May 13 13:55:56 2024 +0800
[feat](Nereids): Reject Commutativity Swap for Nested Loop Joins Affecting
Parallelism (#34639)
---
.../rules/exploration/join/JoinCommute.java| 7 +++
.../apache/doris/planner/NestedLoopJoinNode.java | 6 +-
.../rules/exploration/join/JoinCommuteTest.java| 24 +-
3 files changed, 35 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
index 49c91b92942..73fb853082c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
@@ -25,9 +25,11 @@ import org.apache.doris.nereids.trees.expressions.Not;
import org.apache.doris.nereids.trees.expressions.Slot;
import
org.apache.doris.nereids.trees.expressions.functions.scalar.BitmapContains;
import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.JoinType;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.planner.NestedLoopJoinNode;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.thrift.TRuntimeFilterType;
@@ -66,6 +68,11 @@ public class JoinCommute extends OneExplorationRuleFactory {
// commuting nest loop mark join or left anti mark join is not
supported by be
.whenNot(join -> join.isMarkJoin() &&
(join.getHashJoinConjuncts().isEmpty()
|| join.getJoinType().isLeftAntiJoin()))
+// For a nested loop join, if commutativity causes a join that
could originally be executed
+// in parallel to become non-parallelizable, then we reject
this swap.
+.whenNot(join -> JoinUtils.shouldNestedLoopJoin(join)
+&&
NestedLoopJoinNode.canParallelize(JoinType.toJoinOperator(join.getJoinType()))
+&&
!NestedLoopJoinNode.canParallelize(JoinType.toJoinOperator(join.getJoinType().swap(
.then(join -> {
LogicalJoin newJoin =
join.withTypeChildren(join.getJoinType().swap(),
join.right(), join.left(), null);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
index 983cbfd5884..30c0a2d0394 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
@@ -73,12 +73,16 @@ public class NestedLoopJoinNode extends JoinNodeBase {
tupleIds.addAll(inner.getOutputTupleIds());
}
-public boolean canParallelize() {
+public static boolean canParallelize(JoinOperator joinOp) {
return joinOp == JoinOperator.CROSS_JOIN || joinOp ==
JoinOperator.INNER_JOIN
|| joinOp == JoinOperator.LEFT_OUTER_JOIN || joinOp ==
JoinOperator.LEFT_SEMI_JOIN
|| joinOp == JoinOperator.LEFT_ANTI_JOIN || joinOp ==
JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN;
}
+public boolean canParallelize() {
+return canParallelize(joinOp);
+}
+
public void setJoinConjuncts(List joinConjuncts) {
this.joinConjuncts = joinConjuncts;
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/join/JoinCommuteTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/join/JoinCommuteTest.java
index 20323d108e6..18235b3ce4c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/join/JoinCommuteTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/join/JoinCommuteTest.java
@@ -18,7 +18,9 @@
package org.apache.doris.nereids.rules.exploration.join;
import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
import
Re: [PR] [feat](Nereids): Reject Commutativity Swap for Nested Loop Joins Affecting Parallelism [doris]
jackwener merged PR #34639: URL: https://github.com/apache/doris/pull/34639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) Avoid unnecessary copies in the profile thrift. [doris]
doris-robot commented on PR #34720: URL: https://github.com/apache/doris/pull/34720#issuecomment-2106706157 TPC-DS: Total hot run time: 187370 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit de6bdf54940e3437c5a0ce1aba98112b24ca1ef8, data reload: false query1 933 385 356 356 query2 6444255022772277 query3 6644211 216 211 query4 24345 21288 21293 21288 query5 4149421 425 421 query6 262 173 194 173 query7 4579291 283 283 query8 239 189 195 189 query9 8656234623422342 query10 444 258 240 240 query11 14849 14181 14092 14092 query12 149 94 91 91 query13 1658376 376 376 query14 9760802384908023 query15 256 175 178 175 query16 8014260 252 252 query17 1752555 547 547 query18 2060275 285 275 query19 207 150 152 150 query20 93 83 85 83 query21 195 127 128 127 query22 5001484948734849 query23 34513 33453 33494 33453 query24 6917300228522852 query25 583 379 368 368 query26 703 158 163 158 query27 2154316 330 316 query28 4962204220502042 query29 837 617 620 617 query30 297 152 160 152 query31 983 764 733 733 query32 91 52 55 52 query33 514 252 251 251 query34 886 478 502 478 query35 766 692 658 658 query36 1075929 911 911 query37 106 67 73 67 query38 2849276827532753 query39 1642155515801555 query40 202 126 128 126 query41 43 39 39 39 query42 103 97 103 97 query43 557 552 555 552 query44 1109738 747 738 query45 259 249 241 241 query46 1058741 687 687 query47 1943187518691869 query48 378 296 293 293 query49 953 396 407 396 query50 788 379 392 379 query51 6868674466986698 query52 110 89 96 89 query53 355 287 292 287 query54 532 425 456 425 query55 77 73 72 72 query56 240 227 228 227 query57 1203112511521125 query58 217 204 232 204 query59 3251308931583089 query60 261 231 241 231 query61 92 90 88 88 query62 643 484 483 483 query63 314 286 291 286 query64 8535742474107410 query65 3199310730773077 query66 734 340 349 340 query67 15298 14939 14836 14836 query68 4529537 526 526 query69 453 309 301 301 query70 1153113111571131 query71 367 268 263 263 query72 7762255924012401 query73 700 322 321 321 query74 6456618061616161 query75 3335265926312631 query76 2310996 1002996 query77 415 268 274 268 query78 10636 10257 10126 10126 query79 1245530 520 520 query80 1022451 464 451 query81 537 226 223 223 query82 118996 96 96 query83 257 172 187 172 query84 251 90 90 90 query85 1003391 268 268 query86 370 282 334 282 query87 3267311931423119 query88 2944234023312331 query89 460 397 403 397 query90 1789189 192 189 query91 123 100 97 97 query92 57 50 51 50 query93 1013519 502 502 query94 1004181 184 181 query95 411 315 311 311 query96 574 262 266 262 query97 3182300429592959 query98 240 226 216 216 query99 1214910 909 909 Total cold run time: 270157 ms Total hot run time: 187370 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the problem that DLF cannot be used normally [doris]
CalvinKirs commented on PR #34749: URL: https://github.com/apache/doris/pull/34749#issuecomment-2106702500 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
doris-robot commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106699697 TeamCity be ut coverage result: Function Coverage: 35.66% (8986/25202) Line Coverage: 27.32% (74273/271877) Region Coverage: 26.55% (38386/144583) Branch Coverage: 23.37% (19577/83774) Coverage Report: http://coverage.selectdb-in.cc/coverage/91839fd2456d977a8581700d57a76d4ad9df79b4_91839fd2456d977a8581700d57a76d4ad9df79b4/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): Reject Commutativity Swap for Nested Loop Joins Affecting Parallelism [doris]
github-actions[bot] commented on PR #34639: URL: https://github.com/apache/doris/pull/34639#issuecomment-2106699286 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](status) shoud use add error message method not using index [doris]
yiguolei commented on PR #34754: URL: https://github.com/apache/doris/pull/34754#issuecomment-2106697983 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](status) shoud use add error message method not using index [doris]
doris-robot commented on PR #34754: URL: https://github.com/apache/doris/pull/34754#issuecomment-2106697906 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [bugfix](status) shoud use add error message method not using index [doris]
yiguolei opened a new pull request, #34754: URL: https://github.com/apache/doris/pull/34754 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) Avoid unnecessary copies in the profile thrift. [doris]
doris-robot commented on PR #34720: URL: https://github.com/apache/doris/pull/34720#issuecomment-2106695203 TPC-H: Total hot run time: 41545 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit de6bdf54940e3437c5a0ce1aba98112b24ca1ef8, data reload: false -- Round 1 -- q1 17878 465242684268 q2 2019185 212 185 q3 10535 114611451145 q4 10181 811 838 811 q5 7529274427342734 q6 221 131 132 131 q7 1057596 610 596 q8 9261220821462146 q9 9371686265296529 q10 9108393638413841 q11 470 247 250 247 q12 465 227 220 220 q13 18763 315731763157 q14 271 225 219 219 q15 524 482 481 481 q16 535 404 401 401 q17 1003634 718 634 q18 8603788375547554 q19 5923157015581558 q20 671 328 318 318 q21 5193411840904090 q22 346 285 280 280 Total cold run time: 119927 ms Total hot run time: 41545 ms - Round 2, with runtime_filter_mode=off - q1 4342423642684236 q2 366 268 262 262 q3 3049280727512751 q4 1857161415811581 q5 5314531453125312 q6 207 123 119 119 q7 2237190518841884 q8 3184335633373337 q9 8449847583528352 q10 3914369937213699 q11 579 482 513 482 q12 793 581 585 581 q13 17419 299730312997 q14 299 254 254 254 q15 504 475 475 475 q16 461 406 419 406 q17 1795146315011463 q18 7630776774217421 q19 1686156115881561 q20 1979180817471747 q21 10497 479948144799 q22 580 500 504 500 Total cold run time: 77141 ms Total hot run time: 54219 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](pipeline) Refactor blocking dependencies [doris]
github-actions[bot] commented on PR #34752: URL: https://github.com/apache/doris/pull/34752#issuecomment-2106682943 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](pipeline) Refactor blocking dependencies [doris]
github-actions[bot] commented on PR #34752: URL: https://github.com/apache/doris/pull/34752#issuecomment-2106682914 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): lazy get expression map [doris]
doris-robot commented on PR #34753: URL: https://github.com/apache/doris/pull/34753#issuecomment-2106679097 TPC-DS: Total hot run time: 186290 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 797161beeb0b45defc4a9786f69d15cad1ea168e, data reload: false query1 909 361 345 345 query2 6448242923222322 query3 6658205 212 205 query4 23162 21094 21071 21071 query5 4125413 435 413 query6 251 174 169 169 query7 4588294 296 294 query8 244 190 194 190 query9 8381243124162416 query10 422 251 263 251 query11 14674 14356 14135 14135 query12 132 90 85 85 query13 1650376 355 355 query14 9840849069406940 query15 279 167 169 167 query16 8045261 255 255 query17 1888546 569 546 query18 2008284 266 266 query19 241 147 148 147 query20 94 85 89 85 query21 187 134 123 123 query22 5200487349194873 query23 34332 33579 33690 33579 query24 11873 287327932793 query25 631 377 366 366 query26 1733153 153 153 query27 3018324 327 324 query28 7609208120472047 query29 989 642 592 592 query30 294 147 154 147 query31 974 745 731 731 query32 91 53 54 53 query33 752 247 239 239 query34 1055479 489 479 query35 821 681 653 653 query36 1077909 920 909 query37 141 66 73 66 query38 2885277627712771 query39 1616154115701541 query40 271 121 123 121 query41 42 40 38 38 query42 102 98 95 95 query43 567 532 530 530 query44 1233727 730 727 query45 266 255 250 250 query46 1103725 743 725 query47 1985191918871887 query48 363 301 305 301 query49 1188410 405 405 query50 783 386 393 386 query51 6841657365876573 query52 106 88 89 88 query53 350 285 273 273 query54 980 451 442 442 query55 74 73 70 70 query56 253 226 237 226 query57 1279115611891156 query58 233 225 212 212 query59 3377330532333233 query60 267 246 240 240 query61 106 104 107 104 query62 672 464 463 463 query63 321 285 282 282 query64 9818749074547454 query65 3172313231073107 query66 1386353 349 349 query67 15489 15438 15089 15089 query68 6122546 529 529 query69 549 301 311 301 query70 1128112911661129 query71 454 263 265 263 query72 7666254023882388 query73 744 332 324 324 query74 6571617860296029 query75 3843265626302630 query76 3338950 971 950 query77 620 260 260 260 query78 10756 10215 99959995 query79 3695507 515 507 query80 1292430 424 424 query81 502 222 219 219 query82 103695 100 95 query83 198 164 169 164 query84 271 89 126 89 query85 1465261 257 257 query86 470 292 293 292 query87 3338314631033103 query88 4390243924282428 query89 494 374 385 374 query90 1984188 184 184 query91 121 95 94 94 query92 58 50 47 47 query93 4634505 488 488 query94 1159174 181 174 query95 409 296 297 296 query96 598 273 264 264 query97 3202297229882972 query98 246 220 209 209 query99 1318890 891 890 Total cold run time: 294615 ms Total hot run time: 186290 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
doris-robot commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106678065 TPC-DS: Total hot run time: 187134 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 91839fd2456d977a8581700d57a76d4ad9df79b4, data reload: false query1 910 373 357 357 query2 6438246822942294 query3 6635212 207 207 query4 23178 21164 21364 21164 query5 4100439 418 418 query6 256 186 195 186 query7 4587290 289 289 query8 242 196 200 196 query9 8560245324522452 query10 440 259 249 249 query11 14679 14150 14180 14150 query12 141 93 90 90 query13 1641390 371 371 query14 10570 850377377737 query15 261 172 183 172 query16 7871265 267 265 query17 1767571 567 567 query18 1642290 280 280 query19 234 157 155 155 query20 95 88 88 88 query21 199 131 126 126 query22 5113481549194815 query23 34202 33487 33580 33487 query24 6587286829532868 query25 589 388 384 384 query26 691 158 155 155 query27 1937319 334 319 query28 3784207820602060 query29 905 616 594 594 query30 244 150 160 150 query31 980 730 739 730 query32 85 52 53 52 query33 491 251 249 249 query34 875 474 488 474 query35 769 679 676 676 query36 1057923 920 920 query37 103 65 68 65 query38 2893281227182718 query39 1603157815551555 query40 192 125 127 125 query41 42 38 38 38 query42 101 94 97 94 query43 576 550 552 550 query44 1070727 726 726 query45 266 263 248 248 query46 1064743 720 720 query47 1929188618721872 query48 378 296 304 296 query49 831 386 390 386 query50 765 398 389 389 query51 6765663067346630 query52 110 89 92 89 query53 346 288 280 280 query54 543 439 439 439 query55 75 72 74 72 query56 245 225 231 225 query57 1228114011311131 query58 214 196 202 196 query59 3339319432253194 query60 256 244 239 239 query61 90 87 88 87 query62 599 498 472 472 query63 309 283 288 283 query64 8422738973537353 query65 3130308431073084 query66 774 341 339 339 query67 15496 14933 15079 14933 query68 4572533 533 533 query69 495 311 301 301 query70 1133117011091109 query71 360 281 275 275 query72 7338262323382338 query73 705 332 338 332 query74 6478613261276127 query75 3356268226492649 query76 2541919 938 919 query77 564 285 267 267 query78 10611 10070 10077 10070 query79 2186508 540 508 query80 1059427 440 427 query81 514 224 222 222 query82 661 96 96 96 query83 242 162 164 162 query84 238 85 86 85 query85 1536274 257 257 query86 480 320 280 280 query87 3273311330883088 query88 3990247124472447 query89 483 381 385 381 query90 2047190 186 186 query91 124 99 99 99 query92 63 48 50 48 query93 2689510 506 506 query94 1171184 184 184 query95 390 305 315 305 query96 589 281 266 266 query97 3130298629912986 query98 244 218 216 216 query99 1232891 893 891 Total cold run time: 271003 ms Total hot run time: 187134 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
github-actions[bot] commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106675077 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [optimize](information_schema) optimize the data type of table information_schema.rowsets [doris]
DarvenDuan commented on PR #34739: URL: https://github.com/apache/doris/pull/34739#issuecomment-2106674283 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [optimize](information_schema) optimize the data type of table information_schema.rowsets [doris]
github-actions[bot] commented on PR #34739: URL: https://github.com/apache/doris/pull/34739#issuecomment-2106673980 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): lazy get expression map [doris]
doris-robot commented on PR #34753: URL: https://github.com/apache/doris/pull/34753#issuecomment-2106667878 TPC-H: Total hot run time: 39939 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 797161beeb0b45defc4a9786f69d15cad1ea168e, data reload: false -- Round 1 -- q1 17625 428642484248 q2 2012189 193 189 q3 10501 127611551155 q4 10457 818 784 784 q5 7487274126362636 q6 218 134 132 132 q7 958 531 529 529 q8 9282209620842084 q9 9325666266116611 q10 9702374036903690 q11 473 239 239 239 q12 506 211 215 211 q13 1 295029462946 q14 263 218 225 218 q15 514 476 476 476 q16 507 387 378 378 q17 978 646 764 646 q18 8087736073777360 q19 5244149114901490 q20 655 306 317 306 q21 5053333239643332 q22 362 279 282 279 Total cold run time: 117986 ms Total hot run time: 39939 ms - Round 2, with runtime_filter_mode=off - q1 4396422142164216 q2 374 278 281 278 q3 2988275727242724 q4 1927161616381616 q5 5266527952305230 q6 209 123 123 123 q7 1716137213571357 q8 3185330533183305 q9 8357832282938293 q10 3861369336863686 q11 570 495 498 495 q12 784 606 619 606 q13 16459 294629642946 q14 307 272 259 259 q15 512 477 493 477 q16 456 417 410 410 q17 1763149914571457 q18 7487742273197319 q19 1678154315741543 q20 1979175417571754 q21 4910494149634941 q22 583 507 538 507 Total cold run time: 69767 ms Total hot run time: 53542 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
doris-robot commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106667227 TPC-H: Total hot run time: 41742 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 91839fd2456d977a8581700d57a76d4ad9df79b4, data reload: false -- Round 1 -- q1 17594 424341934193 q2 2019184 201 184 q3 10450 122411481148 q4 10209 787 843 787 q5 7460277427392739 q6 223 134 135 134 q7 1023615 585 585 q8 9242212220812081 q9 9337672767166716 q10 9232392438593859 q11 461 240 244 240 q12 427 236 226 226 q13 18275 321131143114 q14 256 219 224 219 q15 507 472 477 472 q16 505 405 398 398 q17 1003648 679 648 q18 8285777278627772 q19 3518153415501534 q20 635 334 330 330 q21 5182410040754075 q22 340 291 288 288 Total cold run time: 116183 ms Total hot run time: 41742 ms - Round 2, with runtime_filter_mode=off - q1 4554440943154315 q2 375 262 277 262 q3 3192298228832883 q4 1888158515971585 q5 5519545355015453 q6 213 120 122 120 q7 2350195919791959 q8 3252338134313381 q9 8697860186378601 q10 3929385839003858 q11 608 496 510 496 q12 786 632 638 632 q13 15984 317131833171 q14 303 257 260 257 q15 521 468 473 468 q16 477 412 422 412 q17 1767148314801480 q18 7562767374087408 q19 1641156015861560 q20 1944174417691744 q21 7920496947584758 q22 560 493 501 493 Total cold run time: 74042 ms Total hot run time: 55296 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
doris-robot commented on PR #34679: URL: https://github.com/apache/doris/pull/34679#issuecomment-2106655365 TPC-DS: Total hot run time: 187193 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0178bdb538ded706668b74e75aee8cbe193ae9d5, data reload: false query1 926 364 350 350 query2 6447238223072307 query3 6654202 205 202 query4 24000 21189 21093 21093 query5 4184409 410 409 query6 280 173 174 173 query7 4585292 284 284 query8 250 190 195 190 query9 8566240423852385 query10 442 255 252 252 query11 14802 14162 14200 14162 query12 140 93 110 93 query13 1630379 365 365 query14 9054795785297957 query15 266 165 173 165 query16 8169255 265 255 query17 1862584 565 565 query18 2106283 266 266 query19 202 153 150 150 query20 92 85 85 85 query21 194 127 127 127 query22 5077487849054878 query23 34174 33541 33639 33541 query24 11591 295628842884 query25 646 366 360 360 query26 1700165 151 151 query27 2989323 330 323 query28 7274207420452045 query29 969 617 602 602 query30 300 152 153 152 query31 1021738 750 738 query32 93 56 55 55 query33 760 267 265 265 query34 1041480 492 480 query35 813 675 669 669 query36 1071873 901 873 query37 143 67 67 67 query38 2901278227552755 query39 1622155515751555 query40 281 128 126 126 query41 41 36 36 36 query42 107 96 99 96 query43 586 569 532 532 query44 1240740 737 737 query45 273 252 258 252 query46 1088720 722 720 query47 2003188119151881 query48 366 306 295 295 query49 1160413 397 397 query50 779 392 377 377 query51 6800672466436643 query52 102 90 91 90 query53 378 287 285 285 query54 881 435 435 435 query55 76 72 72 72 query56 248 224 220 220 query57 1234116011221122 query58 235 201 207 201 query59 3466310431453104 query60 252 232 245 232 query61 88 89 86 86 query62 660 482 474 474 query63 306 284 280 280 query64 9761743773517351 query65 3142307930713071 query66 1388336 341 336 query67 15505 14923 15168 14923 query68 4558541 553 541 query69 496 303 300 300 query70 1158116411561156 query71 400 279 281 279 query72 7833253323942394 query73 699 325 327 325 query74 6482611161816111 query75 3533263225842584 query76 2915102510231023 query77 388 271 264 264 query78 10633 10112 10139 10112 query79 2574515 522 515 query80 1050452 434 434 query81 500 228 220 220 query82 843 92 92 92 query83 242 164 168 164 query84 240 86 83 83 query85 1382267 367 267 query86 449 308 290 290 query87 3229308130613061 query88 3998232323332323 query89 463 380 393 380 query90 1965186 190 186 query91 127 99 100 99 query92 63 50 50 50 query93 1814525 506 506 query94 1281185 188 185 query95 401 305 306 305 query96 579 272 263 263 query97 3159296129582958 query98 243 217 217 217 query99 1120951 904 904 Total cold run time: 286299 ms Total hot run time: 187193 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [Improvement](set) enable admin_set_frontend_config can apply to all fe [doris]
doris-robot commented on PR #34751: URL: https://github.com/apache/doris/pull/34751#issuecomment-2106652374 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 9edb2fbdfcfde4e36a7be95ebf1ac6f59f0f30ad with default session variables Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 21.8 seconds inserted 1000 Rows, about 458K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](set) enable admin_set_frontend_config can apply to all fe [doris]
doris-robot commented on PR #34751: URL: https://github.com/apache/doris/pull/34751#issuecomment-2106649643 ClickBench: Total hot run time: 30.77 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 9edb2fbdfcfde4e36a7be95ebf1ac6f59f0f30ad, data reload: false query1 0.020.020.02 query2 0.070.020.03 query3 0.240.040.04 query4 1.790.060.07 query5 0.540.530.51 query6 1.240.620.65 query7 0.010.010.01 query8 0.030.020.02 query9 0.520.490.48 query10 0.550.530.54 query11 0.120.090.09 query12 0.110.100.09 query13 0.620.610.60 query14 0.770.780.77 query15 0.770.760.77 query16 0.370.360.39 query17 1.011.011.00 query18 0.220.260.24 query19 1.921.881.86 query20 0.020.010.00 query21 15.52 0.560.57 query22 2.042.301.63 query23 17.16 0.990.96 query24 6.710.741.09 query25 0.330.140.05 query26 0.780.150.16 query27 0.040.040.03 query28 6.600.760.76 query29 12.62 2.402.36 query30 0.550.520.52 query31 2.800.380.37 query32 3.390.490.50 query33 3.053.093.08 query34 15.25 4.824.78 query35 4.884.894.93 query36 1.071.051.06 query37 0.060.050.05 query38 0.030.020.02 query39 0.020.020.01 query40 0.160.140.14 query41 0.060.020.02 query42 0.020.010.02 query43 0.020.020.02 Total cold run time: 104.1 s Total hot run time: 30.77 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): lazy get expression map [doris]
XieJiann commented on PR #34753: URL: https://github.com/apache/doris/pull/34753#issuecomment-2106646548 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): lazy get expression map [doris]
doris-robot commented on PR #34753: URL: https://github.com/apache/doris/pull/34753#issuecomment-2106646494 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [feat](Nereids): lazy get expression map [doris]
XieJiann opened a new pull request, #34753: URL: https://github.com/apache/doris/pull/34753 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
doris-robot commented on PR #34679: URL: https://github.com/apache/doris/pull/34679#issuecomment-2106645621 TPC-H: Total hot run time: 40232 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0178bdb538ded706668b74e75aee8cbe193ae9d5, data reload: false -- Round 1 -- q1 17597 487442684268 q2 2020189 182 182 q3 10574 128011921192 q4 11042 868 822 822 q5 7805273626592659 q6 217 133 132 132 q7 1018605 602 602 q8 9697218321252125 q9 9276669766446644 q10 9525374236873687 q11 468 238 233 233 q12 438 211 219 211 q13 17819 297729832977 q14 265 221 225 221 q15 516 480 470 470 q16 480 387 383 383 q17 967 646 802 646 q18 8173752074867486 q19 4888155114651465 q20 655 307 304 304 q21 5045323241153232 q22 361 291 302 291 Total cold run time: 118846 ms Total hot run time: 40232 ms - Round 2, with runtime_filter_mode=off - q1 4332419642044196 q2 368 261 276 261 q3 2944277127782771 q4 1870159115771577 q5 5307530852865286 q6 212 122 122 122 q7 2202189018831883 q8 3211334733583347 q9 8415841883618361 q10 3918373036153615 q11 596 500 487 487 q12 763 619 604 604 q13 17223 295329362936 q14 283 282 256 256 q15 513 478 474 474 q16 482 407 423 407 q17 1793148814901488 q18 7645758973207320 q19 1656154215731542 q20 1969178217511751 q21 4968478448614784 q22 565 496 496 496 Total cold run time: 71235 ms Total hot run time: 53964 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](set) enable admin_set_frontend_config can apply to all fe [doris]
doris-robot commented on PR #34751: URL: https://github.com/apache/doris/pull/34751#issuecomment-2106645045 TPC-DS: Total hot run time: 204808 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9edb2fbdfcfde4e36a7be95ebf1ac6f59f0f30ad, data reload: false query1 947 405 398 398 query2 6651312029802980 query3 7270215 209 209 query4 20925 18302 18031 18031 query5 19784 650665166506 query6 418 217 232 217 query7 5257307 295 295 query8 264 249 254 249 query9 3150275026722672 query10 441 316 291 291 query11 11306 11029 10883 10883 query12 124 75 72 72 query13 5620687 672 672 query14 18948 13676 13483 13483 query15 361 225 227 225 query16 6427283 260 260 query17 13581492869 869 query18 2281406 403 403 query19 205 149 147 147 query20 87 78 84 78 query21 198 102 93 93 query22 5292517051775170 query23 32337 31934 31810 31810 query24 6791655165066506 query25 505 427 426 426 query26 534 165 161 161 query27 1684302 293 293 query28 6196235923392339 query29 2945275828082758 query30 247 163 165 163 query31 899 752 742 742 query32 69 65 64 64 query33 402 257 244 244 query34 865 462 507 462 query35 1103963 922 922 query36 1210119411211121 query37 88 60 59 59 query38 3072292229312922 query39 1367134013151315 query40 202 97 95 95 query41 38 37 36 36 query42 87 77 82 77 query43 682 641 741 641 query44 1120703 716 703 query45 242 231 231 231 query46 1218992 977 977 query47 1817191516931693 query48 998 703 690 690 query49 614 380 363 363 query50 857 598 611 598 query51 4769463946874639 query52 94 71 85 71 query53 455 330 321 321 query54 2645244724872447 query55 85 78 84 78 query56 249 226 218 218 query57 1229114110841084 query58 221 202 209 202 query59 4518395540333955 query60 228 209 199 199 query61 89 85 86 85 query62 841 455 470 455 query63 490 333 342 333 query64 2477151914391439 query65 3641356735693567 query66 754 376 376 376 query67 15784 15966 15747 15747 query68 8397646 659 646 query69 579 350 365 350 query70 1791150016681500 query71 400 315 319 315 query72 6483350834733473 query73 752 328 337 328 query74 6326587158735871 query75 4650365137123651 query76 4783117611871176 query77 654 254 262 254 query78 12768 11798 11653 11653 query79 7759656 639 639 query80 1010395 408 395 query81 500 231 229 229 query82 161796 93 93 query83 168 142 140 140 query84 253 69 70 69 query85 900 300 296 296 query86 339 298 307 298 query87 3240300530823005 query88 4930235723422342 query89 340 297 291 291 query90 1890205 211 205 query91 171 149 134 134 query92 58 53 61 53 query93 3917569 586 569 query94 766 202 206 202 query95 1092106510481048 query96 650 343 328 328 query97 6563634764296347 query98 185 180 170 170 query99 2930908 914 908 Total cold run time: 310933 ms Total hot run time: 204808 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [fix](prepare) fix datetimev2 return err when binary_row_format [doris]
doris-robot commented on PR #34662: URL: https://github.com/apache/doris/pull/34662#issuecomment-2106644302 TPC-DS: Total hot run time: 187887 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit a1014c0c52335c44bbda65032d89ac688bb01f08, data reload: false query1 924 363 353 353 query2 6450248923792379 query3 6655209 213 209 query4 23093 21241 21124 21124 query5 4147417 431 417 query6 269 170 194 170 query7 4575295 292 292 query8 245 195 194 194 query9 8769241724092409 query10 440 254 267 254 query11 14799 14326 14223 14223 query12 130 95 88 88 query13 1634372 374 372 query14 9755780478377804 query15 232 168 177 168 query16 7727253 262 253 query17 1858584 536 536 query18 1384276 276 276 query19 204 146 151 146 query20 95 88 86 86 query21 203 131 132 131 query22 5043495049294929 query23 34267 33868 33683 33683 query24 11670 295929742959 query25 630 365 376 365 query26 1525161 157 157 query27 2701314 322 314 query28 7403206620592059 query29 948 611 630 611 query30 289 156 150 150 query31 998 808 744 744 query32 87 53 56 53 query33 739 243 258 243 query34 1038474 490 474 query35 806 684 657 657 query36 1077904 933 904 query37 130 69 68 68 query38 2881278527722772 query39 1630154115621541 query40 274 125 126 125 query41 42 38 39 38 query42 107 95 100 95 query43 595 568 554 554 query44 1236721 744 721 query45 265 257 258 257 query46 1075721 704 704 query47 1942189519191895 query48 375 291 299 291 query49 1145395 430 395 query50 772 388 396 388 query51 6850667366566656 query52 101 92 94 92 query53 355 299 297 297 query54 1083433 434 433 query55 75 73 72 72 query56 249 225 237 225 query57 1246113111301130 query58 235 205 216 205 query59 3543318134013181 query60 258 243 245 243 query61 90 89 87 87 query62 655 500 464 464 query63 312 286 316 286 query64 9802738173217321 query65 3259316931323132 query66 1296348 354 348 query67 15393 14954 15138 14954 query68 4594532 538 532 query69 478 310 318 310 query70 1195115611681156 query71 423 274 288 274 query72 7247260623592359 query73 703 321 322 321 query74 6574612761786127 query75 3362264226032603 query76 2764926 916 916 query77 413 272 273 272 query78 10644 10180 10216 10180 query79 1488516 516 516 query80 1062446 442 442 query81 504 222 220 220 query82 957 93 96 93 query83 265 162 165 162 query84 232 85 86 85 query85 1242327 339 327 query86 381 305 309 305 query87 3277310430953095 query88 3153243124212421 query89 466 375 396 375 query90 1925187 186 186 query91 127 100 96 96 query92 58 50 49 49 query93 1325510 494 494 query94 1211182 185 182 query95 384 302 302 302 query96 587 274 266 266 query97 3152299229592959 query98 234 223 213 213 query99 1167911 893 893 Total cold run time: 281674 ms Total hot run time: 187887 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
github-actions[bot] commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106641598 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
xinyiZzz commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106636931 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](prepare) fix datetimev2 return err when binary_row_format [doris]
doris-robot commented on PR #34662: URL: https://github.com/apache/doris/pull/34662#issuecomment-2106633899 TPC-H: Total hot run time: 39876 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit a1014c0c52335c44bbda65032d89ac688bb01f08, data reload: false -- Round 1 -- q1 17978 526043874387 q2 2642188 197 188 q3 10988 119111921191 q4 10350 739 742 739 q5 7546271626042604 q6 213 127 127 127 q7 1038614 588 588 q8 9421210620402040 q9 8978655665046504 q10 8968365337433653 q11 476 242 233 233 q12 458 221 211 211 q13 17778 297929442944 q14 246 220 218 218 q15 532 478 474 474 q16 509 393 377 377 q17 968 631 673 631 q18 8106747373517351 q19 5824156115371537 q20 666 312 313 312 q21 5114389132863286 q22 346 281 286 281 Total cold run time: 119145 ms Total hot run time: 39876 ms - Round 2, with runtime_filter_mode=off - q1 4389424441864186 q2 386 278 272 272 q3 3027274927492749 q4 1899163216041604 q5 5281525952915259 q6 205 122 121 121 q7 2239190218751875 q8 3232335733303330 q9 8418834383808343 q10 3897366736753667 q11 593 484 493 484 q12 760 588 589 588 q13 16321 297830012978 q14 280 256 260 256 q15 519 478 471 471 q16 477 413 420 413 q17 1795149014721472 q18 7769759472897289 q19 1651151315811513 q20 1948177917581758 q21 4783485950084859 q22 554 520 503 503 Total cold run time: 70423 ms Total hot run time: 53990 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](set) enable admin_set_frontend_config can apply to all fe [doris]
doris-robot commented on PR #34751: URL: https://github.com/apache/doris/pull/34751#issuecomment-2106633589 TPC-H: Total hot run time: 49564 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9edb2fbdfcfde4e36a7be95ebf1ac6f59f0f30ad, data reload: false -- Round 1 -- q1 17658 436842974297 q2 2054155 143 143 q3 10280 189118981891 q4 10383 124613241246 q5 8451386838883868 q6 235 124 122 122 q7 2015157516301575 q8 9495271926892689 q9 10891 10341 10421 10341 q10 8616352735013501 q11 429 232 243 232 q12 460 295 306 295 q13 18338 395740483957 q14 354 331 335 331 q15 505 452 466 452 q16 649 562 578 562 q17 1130983 963 963 q18 7314694868076807 q19 1684155414921492 q20 541 309 314 309 q21 4431412241464122 q22 489 369 395 369 Total cold run time: 116402 ms Total hot run time: 49564 ms - Round 2, with runtime_filter_mode=off - q1 4293429042804280 q2 320 220 226 220 q3 4189415341394139 q4 2753274627602746 q5 7160716371407140 q6 233 116 116 116 q7 3205291228682868 q8 4364442345184423 q9 17356 17009 17252 17009 q10 4236423742634237 q11 735 687 726 687 q12 1028857 847 847 q13 6686376237133713 q14 444 434 423 423 q15 511 467 450 450 q16 733 677 677 677 q17 3815392638093809 q18 8916884387938793 q19 1714172316821682 q20 2414213421432134 q21 8522868385838583 q22 1083980 975 975 Total cold run time: 84710 ms Total hot run time: 79951 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) Avoid unnecessary copies in the profile thrift. [doris]
Mryange commented on PR #34720: URL: https://github.com/apache/doris/pull/34720#issuecomment-2106628396 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](pipeline) Refactor blocking dependencies [doris]
github-actions[bot] commented on code in PR #34752:
URL: https://github.com/apache/doris/pull/34752#discussion_r1597854622
##
be/src/pipeline/pipeline_task.cpp:
##
@@ -197,6 +198,35 @@
_task_queue = task_queue;
}
+bool PipelineTask::_is_blocked() {
+// `_dry_run = true` means we do not need data from source operator.
+if (!_dry_run) {
+for (int i = _read_dependencies.size() - 1; i >= 0; i--) {
+// `_read_dependencies` is organized according to operators. For
each operator, running condition is met iff all dependencies are ready.
+for (auto* dep : _read_dependencies[i]) {
+_blocked_dep = dep->is_blocked_by(this);
+if (_blocked_dep != nullptr) {
+_blocked_dep->start_watcher();
+return true;
+}
+}
+// If all dependencies are ready for this operator, we can execute
this task if no datum is needed from upstream operators.
+if (!_operators[i]->need_more_input_data(_state)) {
+break;
+}
+}
+}
+
+for (auto* op_dep : _write_dependencies) {
+_blocked_dep = op_dep->is_blocked_by(this);
+if (_blocked_dep != nullptr) {
+_blocked_dep->start_watcher();
+return true;
+}
+}
+return false;
+}
+
Status PipelineTask::execute(bool* eos) {
Review Comment:
warning: function 'execute' has cognitive complexity of 58 (threshold 50)
[readability-function-cognitive-complexity]
```cpp
Status PipelineTask::execute(bool* eos) {
^
```
Additional context
**be/src/pipeline/pipeline_task.cpp:234:** +1, including nesting penalty of
0, nesting level increased to 1
```cpp
if (_eos) {
^
```
**be/src/pipeline/pipeline_task.cpp:253:** +1, including nesting penalty of
0, nesting level increased to 1
```cpp
if (has_dependency() || _runtime_filter_blocked_dependency() != nullptr)
{
^
```
**be/src/pipeline/pipeline_task.cpp:253:** +1
```cpp
if (has_dependency() || _runtime_filter_blocked_dependency() != nullptr)
{
^
```
**be/src/pipeline/pipeline_task.cpp:257:** +1, including nesting penalty of
0, nesting level increased to 1
```cpp
if (!_opened) {
^
```
**be/src/pipeline/pipeline_task.cpp:258:** +2, including nesting penalty of
1, nesting level increased to 2
```cpp
{ RETURN_IF_ERROR(_open()); }
^
```
**be/src/common/status.h:570:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/pipeline/pipeline_task.cpp:258:** +3, including nesting penalty of
2, nesting level increased to 3
```cpp
{ RETURN_IF_ERROR(_open()); }
^
```
**be/src/common/status.h:572:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/pipeline/pipeline_task.cpp:261:** +1, including nesting penalty of
0, nesting level increased to 1
```cpp
while (!_fragment_context->is_canceled()) {
^
```
**be/src/pipeline/pipeline_task.cpp:262:** +2, including nesting penalty of
1, nesting level increased to 2
```cpp
if (_is_blocked()) {
^
```
**be/src/pipeline/pipeline_task.cpp:269:** +2, including nesting penalty of
1, nesting level increased to 2
```cpp
if (_fragment_context->is_canceled()) {
^
```
**be/src/pipeline/pipeline_task.cpp:273:** +2, including nesting penalty of
1, nesting level increased to 2
```cpp
if (time_spent > THREAD_TIME_SLICE) {
^
```
**be/src/pipeline/pipeline_task.cpp:281:** +2, including nesting penalty of
1, nesting level increased to 2
```cpp
if (should_revoke_memory(_state, sink_revocable_mem_size)) {
^
```
**be/src/pipeline/pipeline_task.cpp:282:** +3, including nesting penalty of
2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_sink->revoke_memory(_state));
^
```
**be/src/common/status.h:570:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/pipeline/pipeline_task.cpp:282:** +4, including nesting penalty of
3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_sink->revoke_memory(_state));
^
```
**be/src/common/status.h:572:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/pipeline/pipeline_task.cpp:288:** +2, including nesting penalty of
1, nesting level increased to 2
```cpp
if (!_dry_run) {
^
```
**be/src/pipeline/pipeline_task.cpp:292:** +3, including nesting penalty of
2, nesting
Re: [PR] [Fix](auto-partition) fix auto partition load lost data in multi sender [doris]
HappenLee commented on code in PR #34740:
URL: https://github.com/apache/doris/pull/34740#discussion_r1597853085
##
be/src/vec/sink/writer/vtablet_writer.cpp:
##
@@ -359,22 +357,23 @@ Status VNodeChannel::init(RuntimeState* state) {
// add block closure
// Has to using value to capture _task_exec_ctx because tablet writer may
destroyed during callback.
_send_block_callback =
WriteBlockCallback::create_shared();
-_send_block_callback->addFailedHandler([&, task_exec_ctx =
_task_exec_ctx](bool is_last_rpc) {
-auto ctx_lock = task_exec_ctx.lock();
-if (ctx_lock == nullptr) {
-return;
-}
-_add_block_failed_callback(is_last_rpc);
-});
+_send_block_callback->addFailedHandler(
+[&, task_exec_ctx = _task_exec_ctx](bool is_last_rpc, bool
first_stage_close) {
Review Comment:
move `first_stage_close` to exec_ctx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) Avoid unnecessary copies in the profile thrift. [doris]
github-actions[bot] commented on PR #34720: URL: https://github.com/apache/doris/pull/34720#issuecomment-2106609897 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](pipeline) Refactor blocking dependencies [doris]
Gabriel39 commented on PR #34752: URL: https://github.com/apache/doris/pull/34752#issuecomment-2106609741 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](pipeline) Refactor blocking dependencies [doris]
doris-robot commented on PR #34752: URL: https://github.com/apache/doris/pull/34752#issuecomment-2106609662 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [refactor](pipeline) Refactor blocking dependencies [doris]
Gabriel39 opened a new pull request, #34752: URL: https://github.com/apache/doris/pull/34752 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](Row store) support column group with store row format for partial columns of table [doris]
xiaokang commented on code in PR #34089:
URL: https://github.com/apache/doris/pull/34089#discussion_r1597834122
##
be/src/olap/tablet_schema.h:
##
@@ -213,6 +215,10 @@ class TabletColumn {
// Use shared_ptr for reuse and reducing column memory usage
std::vector _sparse_cols;
size_t _num_sparse_columns = 0;
+
+// groups of this column belongs to
+// contains column ids of group, and encode the group of columns into row
store
+std::vector _group_ids;
Review Comment:
_row_store_group_ids
##
be/src/olap/rowset/segment_v2/segment_writer.cpp:
##
@@ -800,9 +792,8 @@ Status SegmentWriter::append_block(const vectorized::Block*
block, size_t row_po
<< ", _column_writers.size()=" << _column_writers.size();
// Row column should be filled here when it's a directly write from
memtable
// or it's schema change write(since column data type maybe changed, so we
should reubild)
-if (_tablet_schema->store_row_column() &&
-(_opts.write_type == DataWriteType::TYPE_DIRECT ||
- _opts.write_type == DataWriteType::TYPE_SCHEMA_CHANGE)) {
+if (_opts.write_type == DataWriteType::TYPE_DIRECT ||
Review Comment:
check partial
##
be/src/service/point_query_executor.cpp:
##
@@ -47,16 +52,103 @@
#include "vec/data_types/serde/data_type_serde.h"
#include "vec/exprs/vexpr.h"
#include "vec/exprs/vexpr_context.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/exprs/vslot_ref.h"
#include "vec/jsonb/serialize.h"
#include "vec/sink/vmysql_result_writer.cpp"
#include "vec/sink/vmysql_result_writer.h"
namespace doris {
-Reusable::~Reusable() {}
+Reusable::~Reusable() = default;
+
+// Function to find the best matching row store column from the input tuple
descriptor
+// It returns the row store column's unique id with column groups that matches
the most slots from the input tuple descriptor.
+// If no match return -1
+static int pick_row_store(const TabletSchema& schema, const
std::vector& slots) {
+std::unordered_set input_slots_cid_set;
+for (auto* slot : slots) {
+input_slots_cid_set.insert(slot->col_unique_id());
+}
+int max_match = 0;
+int result_col_unique_id = -1;
+int smallest_size = INT_MAX;
+
+for (const auto& col : schema.columns()) {
+if (!col->is_row_store_column()) {
+continue;
+}
+int current_match = 0;
+// The full column group is considered a full match.
+if (col->group_ids().empty()) {
+current_match = input_slots_cid_set.size();
+}
+// Count how many elements in the current column group are in the
input tuple slots.
+for (int num : col->group_ids()) {
+if (input_slots_cid_set.contains(num)) {
+current_match++;
+}
+}
+
+// Update the best matching set if the current set has more matches,
+// or if it has the same number of matches but is smaller.
+// The smaller set is preferred because it needs read less data.
+if (current_match > 0 &&
+(current_match > max_match ||
+ (current_match == max_match && col->group_ids().size() <
smallest_size))) {
+max_match = current_match;
+result_col_unique_id = col->unique_id();
+smallest_size = col->group_ids().empty() ? INT_MAX :
col->group_ids().size();
+}
+}
+return result_col_unique_id;
+}
+
+static void get_missing_and_include_cids(const TabletSchema& schema,
+ const std::vector&
slots,
+ int target_rs_column_id,
+ std::unordered_set& missing_cids,
+ std::unordered_set&
include_cids) {
+missing_cids.clear();
+include_cids.clear();
+for (auto* slot : slots) {
+missing_cids.insert(slot->col_unique_id());
+}
+if (target_rs_column_id == -1) {
+// no row store columns
+return;
+}
+const TabletColumn& target_rs_column =
schema.column_by_uid(target_rs_column_id);
+DCHECK(target_rs_column.is_row_store_column());
+// The full column group is considered a full match, thus no missing cids
+if (target_rs_column.group_ids().empty()) {
Review Comment:
consider that if it's right after light schema chage
##
be/src/olap/rowset/segment_v2/segment_writer.cpp:
##
@@ -576,11 +571,8 @@ Status
SegmentWriter::append_block_with_partial_content(const vectorized::Block*
RETURN_IF_ERROR(fill_missing_columns(mutable_full_columns,
use_default_or_null_flag,
has_default_or_nullable,
segment_start_pos));
full_block.set_columns(std::move(mutable_full_columns));
-// row column should be filled here
-if (_tablet_schema->store_row_column()) {
-// convert block to row store
Re: [PR] [fix](kill_sql) support force cancel sql [doris]
Yulei-Yang closed pull request #33235: [fix](kill_sql) support force cancel sql URL: https://github.com/apache/doris/pull/33235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature][external catalog/lakesoul] support lakesoul catalog [doris]
github-actions[bot] commented on PR #32164: URL: https://github.com/apache/doris/pull/32164#issuecomment-2106598308 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)Mtmv rollup [doris]
zddr commented on PR #31812: URL: https://github.com/apache/doris/pull/31812#issuecomment-2106596831 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (0a102afc478 -> 4f1a88481a0)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 0a102afc478 [Chore](topn) add TTopnFilterContext (#34718) add 4f1a88481a0 [fix](Nereids) fix sql cache return old value when truncate partition (#34698) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/common/Config.java | 8 - .../main/java/org/apache/doris/catalog/Env.java| 4 +-- .../doris/common/NereidsSqlCacheManager.java | 38 +++--- .../org/apache/doris/qe/cache/CacheAnalyzer.java | 2 +- .../cache/parse_sql_from_sql_cache.groovy | 38 +- 5 files changed, 65 insertions(+), 25 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) fix sql cache return old value when truncate partition [doris]
morrySnow merged PR #34698: URL: https://github.com/apache/doris/pull/34698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
xinyiZzz commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106593800 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) fix sql cache return old value when truncate partition [doris]
morrySnow commented on code in PR #34698:
URL: https://github.com/apache/doris/pull/34698#discussion_r1597843033
##
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##
@@ -1314,12 +1314,18 @@ public class Config extends ConfigBase {
* Minimum interval between last version when caching results,
* This parameter distinguishes between offline and real-time updates
*/
+@ConfField(mutable = true, masterOnly = false)
+public static int cache_last_version_interval_second = 30;
Review Comment:
add description in annotation
##
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##
@@ -1314,12 +1314,18 @@ public class Config extends ConfigBase {
* Minimum interval between last version when caching results,
* This parameter distinguishes between offline and real-time updates
*/
+@ConfField(mutable = true, masterOnly = false)
+public static int cache_last_version_interval_second = 30;
+
+/**
+ * Expire sql sql in frontend time
+ */
@ConfField(
mutable = true,
masterOnly = false,
callbackClassString =
"org.apache.doris.common.NereidsSqlCacheManager$UpdateConfig"
)
-public static int cache_last_version_interval_second = 30;
+public static int expire_sql_cache_in_fe_second = 300;
Review Comment:
add description in annotation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
github-actions[bot] commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106592998 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
github-actions[bot] commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106592962 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](set) enable admin_set_frontend_config can apply to all fe [doris]
Yulei-Yang commented on PR #34751: URL: https://github.com/apache/doris/pull/34751#issuecomment-2106592187 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
github-actions[bot] commented on PR #34750:
URL: https://github.com/apache/doris/pull/34750#issuecomment-2106591712
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/9057009698;) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In build.sh line 362:
BUILD_TYPE_LOWWER=`echo "${BUILD_TYPE}" | tr '[:upper:]' '[:lower:]'`
^-- SC2006 (style): Use $(...) notation instead of legacy
backticks `...`.
Did you mean:
BUILD_TYPE_LOWWER=$(echo "${BUILD_TYPE}" | tr '[:upper:]' '[:lower:]')
For more information:
https://www.shellcheck.net/wiki/SC2006 -- Use $(...) notation instead of
le...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- build.sh.orig
+++ build.sh
@@ -359,7 +359,7 @@
USE_MEM_TRACKER='OFF'
fi
fi
-BUILD_TYPE_LOWWER=`echo "${BUILD_TYPE}" | tr '[:upper:]' '[:lower:]'`
+BUILD_TYPE_LOWWER=$(echo "${BUILD_TYPE}" | tr '[:upper:]' '[:lower:]')
if [[ "${BUILD_TYPE_LOWWER}" == "asan" ]]; then
USE_JEMALLOC='OFF'
elif [[ -z "${USE_JEMALLOC}" ]]; then
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](set) enable admin_set_frontend_config can apply to all fe [doris]
doris-robot commented on PR #34751: URL: https://github.com/apache/doris/pull/34751#issuecomment-2106591494 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Improvement](set) enable admin_set_frontend_config can apply to all fe [doris]
Yulei-Yang opened a new pull request, #34751: URL: https://github.com/apache/doris/pull/34751 ## Proposed changes bp: #34685 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
xinyiZzz commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106591162 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
doris-robot commented on PR #34750: URL: https://github.com/apache/doris/pull/34750#issuecomment-2106591132 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](memory) Fix BE compile in Mac with Jemalloc [doris]
xinyiZzz opened a new pull request, #34750: URL: https://github.com/apache/doris/pull/34750 ## Proposed changes 1. Fix DORIS_THIRDPARTY 2. Fix BUILD_TYPE case conversion 3. Mac default disable USE_JEMALLOC_HOOK ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](auto-partition) fix auto partition load lost data in multi sender [doris]
HappenLee commented on code in PR #34740:
URL: https://github.com/apache/doris/pull/34740#discussion_r1597841125
##
be/src/vec/sink/writer/vtablet_writer.cpp:
##
@@ -969,17 +980,50 @@ void VNodeChannel::_close_check() {
CHECK(_cur_mutable_block == nullptr) << name();
}
+void VNodeChannel::close_first_stage() {
+auto st = none_of({_cancelled, _eos_is_produced});
+if (!st.ok()) {
+return;
Review Comment:
Add warning log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
github-actions[bot] commented on code in PR #34679:
URL: https://github.com/apache/doris/pull/34679#discussion_r1597838978
##
be/src/io/fs/s3_file_writer.cpp:
##
@@ -248,7 +242,7 @@ Status S3FileWriter::close() {
}
Status S3FileWriter::appendv(const Slice* data, size_t data_cnt) {
Review Comment:
warning: function 'appendv' has cognitive complexity of 65 (threshold 50)
[readability-function-cognitive-complexity]
```cpp
Status S3FileWriter::appendv(const Slice* data, size_t data_cnt) {
^
```
Additional context
**be/src/io/fs/s3_file_writer.cpp:244:** +1, including nesting penalty of 0,
nesting level increased to 1
```cpp
if (closed() != FileWriterState::OPEN) [[unlikely]] {
^
```
**be/src/io/fs/s3_file_writer.cpp:249:** +1, including nesting penalty of 0,
nesting level increased to 1
```cpp
DBUG_EXECUTE_IF("s3_file_writer::appendv",
^
```
**be/src/util/debug_points.h:35:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (UNLIKELY(config::enable_debug_points)) {
\
^
```
**be/src/io/fs/s3_file_writer.cpp:249:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
DBUG_EXECUTE_IF("s3_file_writer::appendv",
^
```
**be/src/util/debug_points.h:37:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (dp) {
\
^
```
**be/src/io/fs/s3_file_writer.cpp:251:** +1, including nesting penalty of 0,
nesting level increased to 1
```cpp
for (size_t i = 0; i < data_cnt; i++) {
^
```
**be/src/io/fs/s3_file_writer.cpp:253:** +2, including nesting penalty of 1,
nesting level increased to 2
```cpp
for (size_t pos = 0, data_size_to_append = 0; pos < data_size; pos
+= data_size_to_append) {
^
```
**be/src/io/fs/s3_file_writer.cpp:254:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if (_failed) {
^
```
**be/src/io/fs/s3_file_writer.cpp:257:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if (!_pending_buf) {
^
```
**be/src/io/fs/s3_file_writer.cpp:261:** nesting level increased to 4
```cpp
[part_num = _cur_part_num,
this](UploadFileBuffer& buf) {
^
```
**be/src/io/fs/s3_file_writer.cpp:265:** nesting level increased to 4
```cpp
.set_sync_after_complete_task([this, part_num =
_cur_part_num](Status s) {
^
```
**be/src/io/fs/s3_file_writer.cpp:267:** +5, including nesting penalty of 4,
nesting level increased to 5
```cpp
if (!s.ok()) [[unlikely]] {
^
```
**be/src/io/fs/s3_file_writer.cpp:281:** nesting level increased to 4
```cpp
.set_is_cancelled([this]() { return _failed.load();
});
^
```
**be/src/io/fs/s3_file_writer.cpp:282:** +4, including nesting penalty of 3,
nesting level increased to 4
```cpp
if (_cache_builder != nullptr) {
^
```
**be/src/io/fs/s3_file_writer.cpp:294:** +4, including nesting penalty of 3,
nesting level increased to 4
```cpp
RETURN_IF_ERROR(builder.build(&_pending_buf));
^
```
**be/src/common/status.h:570:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/io/fs/s3_file_writer.cpp:294:** +5, including nesting penalty of 4,
nesting level increased to 5
```cpp
RETURN_IF_ERROR(builder.build(&_pending_buf));
^
```
**be/src/common/status.h:572:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/io/fs/s3_file_writer.cpp:303:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
RETURN_IF_ERROR(_pending_buf->append_data(
^
```
**be/src/common/status.h:570:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/io/fs/s3_file_writer.cpp:303:** +4, including nesting penalty of 3,
nesting level increased to 4
```cpp
RETURN_IF_ERROR(_pending_buf->append_data(
^
```
**be/src/common/status.h:572:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/io/fs/s3_file_writer.cpp:309:** +3, including nesting penalty of 2,
nesting level increased to 3
```cpp
if
Re: [PR] [Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the problem that DLF cannot be used normally [doris]
CalvinKirs commented on PR #34749: URL: https://github.com/apache/doris/pull/34749#issuecomment-2106582378 Oops...let's run the test later. It seems that the jar package has not completely completed the release process. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: Add workload system table (#639)
This is an automated email from the ASF dual-hosted git repository.
wangbo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 7efb88a3a5 Add workload system table (#639)
7efb88a3a5 is described below
commit 7efb88a3a5608af98aeea45cd5e490dbab4f5376
Author: wangbo
AuthorDate: Mon May 13 11:49:06 2024 +0800
Add workload system table (#639)
---
.../resource-admin/workload-system-table.md| 230 +
.../resource-admin/workload-system-table.md| 230 +
.../resource-admin/workload-system-table.md| 230 +
sidebars.json | 1 +
.../resource-admin/workload-system-table.md| 230 +
versioned_sidebars/version-2.1-sidebars.json | 1 +
6 files changed, 922 insertions(+)
diff --git a/docs/admin-manual/resource-admin/workload-system-table.md
b/docs/admin-manual/resource-admin/workload-system-table.md
new file mode 100644
index 00..3e8bd9e82e
--- /dev/null
+++ b/docs/admin-manual/resource-admin/workload-system-table.md
@@ -0,0 +1,230 @@
+---
+{
+"title": "Workload System Table",
+"language": "en"
+}
+---
+
+
+
+## Backgroup
+Doris supports analyzing the resource usage of running workloads through the
Workload system table, commonly used in the following scenarios:
+1. View the resource usage of Workload Group in the cluster, including CPU and
memory.
+2. View the TopN SQL with the highest resource usage in the current cluster.
+3. View the queue status of Workload Groups in the cluster
+
+Users can query this information by submitting SQL, identify the Workload
Group or SQL with high resource usage in the system, and perform corresponding
processing.
+
+## Workload System Table Introduction
+At present, the system tables are in database ```information_schema```.
+### active_queries
+The ```active_queries``` table records the execution information of the
current query on FE, and the detailed information of the fields is as follows：
+* query_id, query's id
+* query_start_time, the start time of the query execution; If query queues, it
represents the time when execution starts after the queue ends
+* query_time_ms, the query time, in milliseconds
+* workload_group_id, workload group's id
+* database, sql's database
+* frontend_instance, the FE which query is submitted
+* queue_start_time, if query queues, it means the time that query begins to
queue
+* queue_end_time, if query queues, it means the time that query ends to queue
+* query_status, query status, it has two value RUNNING and QUEUED, RUNNIG
means query is running; QUEUED means query is queued.
+* sql, sql content
+
+### backend_active_tasks
+A query is usually divided into multiple fragments to be executed on multiple
BEs, and ```backend_active_tasks``` table represent the total amount of CPU and
memory resources used by a query on a single BE. If this query has multiple
concurrency and fragments on a single BE, it will also be summarized into one
row of data.
+The detailed information of the fields is as follows:
+* be_id, BE's id
+* fe_host, it represents which FE this query was submitted from
+* query_id, query's id
+* task_time_ms, query execution time on the current BE, in milliseconds
+* task_cpu_time_ms，query cpu time on the current BE, in milliseconds
+* scan_rows, the number of rows scanned on the current BE. If multiple tables
are scanned, it is the cumulative value of multiple tables
+* scan_bytes, the number of bytes scanned on the current BE. If multiple
tables are scanned, it is the cumulative value of multiple tables
+* be_peak_memory_bytes, the peak memory usage on the current BE, in bytes
+* current_used_memory_bytes, the amount of memory currently in use on the BE,
in bytes
+* shuffle_send_bytes, the number of bytes sent as shuffle clients at the BE
+* shuffle_send_rows, the number of rows sent as shuffle clients at the BE
+
+## Basic Usage
+1. TopN resource usage sql
+```
+select
+t2.query_id,
+t2.workload_group_id,
+t2.`database`,
+t1.cpu_time,
+t1.mem_used,
+t2.`sql`
+from
+(select query_id, sum(task_cpu_time_ms) as
cpu_time,sum(current_used_memory_bytes) as mem_used from backend_active_tasks
group by query_id)
+t1 left join active_queries t2
+on t1.query_id = t2.query_id
+order by cpu_time desc, mem_used desc limit 10;
+```
+
+2. TopN resource usage sql on BE
+```
+select
+t2.query_id,
+t2.workload_group_id,
+t2.`database`,
+t1.cpu_time,
+t1.mem_used,
+t2.`sql`
+from
+(select query_id, sum(task_cpu_time_ms) as
cpu_time,sum(current_used_memory_bytes) as mem_used
+from backend_active_tasks where be_id=12345 group by
Error while running notifications feature from .asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [docs]Add workload system table [doris-website]
wangbo merged PR #639: URL: https://github.com/apache/doris-website/pull/639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the problem that DLF cannot be used normally [doris]
github-actions[bot] commented on PR #34749: URL: https://github.com/apache/doris/pull/34749#issuecomment-2106580271 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the problem that DLF cannot be used normally [doris]
github-actions[bot] commented on PR #34749: URL: https://github.com/apache/doris/pull/34749#issuecomment-2106580253 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the problem that DLF cannot be used normally [doris]
morningman commented on PR #34749: URL: https://github.com/apache/doris/pull/34749#issuecomment-2106579223 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
ByteYue commented on PR #34679: URL: https://github.com/apache/doris/pull/34679#issuecomment-2106579101 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)Mtmv rollup [doris]
zddr commented on code in PR #31812:
URL: https://github.com/apache/doris/pull/31812#discussion_r1597816207
##
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescRollUpGenerator.java:
##
@@ -0,0 +1,140 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mtmv;
+
+import org.apache.doris.analysis.PartitionKeyDesc;
+import org.apache.doris.analysis.PartitionValue;
+import org.apache.doris.catalog.PartitionType;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.mtmv.MTMVPartitionInfo.MTMVPartitionType;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Roll up some partitions into one partition
+ */
+public class MTMVRelatedPartitionDescRollUpGenerator implements
MTMVRelatedPartitionDescGeneratorService {
+
+@Override
+public void apply(MTMVPartitionInfo mvPartitionInfo, Map
mvProperties,
+RelatedPartitionDescResult lastResult) throws AnalysisException {
+if (mvPartitionInfo.getPartitionType() != MTMVPartitionType.EXPR) {
+return;
+}
+MTMVRelatedTableIf relatedTable = mvPartitionInfo.getRelatedTable();
+PartitionType partitionType = relatedTable.getPartitionType();
+if (partitionType == PartitionType.RANGE) {
+lastResult.setDescs(rollUpRange(lastResult.getDescs(),
mvPartitionInfo));
+} else if (partitionType == PartitionType.LIST) {
+lastResult.setDescs(rollUpList(lastResult.getDescs(),
mvPartitionInfo, mvProperties));
+} else {
+throw new AnalysisException("only RANGE/LIST partition support
roll up");
+}
+}
+
+/**
+ * when related table has 3 partitions:(20200101),(20200102),(20200201)
+ *
+ * if expr is `date_trunc(month)`
+ * then,MTMV will have 2 partitions (20200101,20200102),(20200201)
+ *
+ * if expr is `date_trunc(year)`
+ * then,MTMV will have 1 partitions (20200101,20200102,20200201)
+ *
+ * @param relatedPartitionDescs
+ * @param mvPartitionInfo
+ * @return
+ * @throws AnalysisException
+ */
+public Map> rollUpList(Map> relatedPartitionDescs,
+MTMVPartitionInfo mvPartitionInfo, Map
mvProperties) throws AnalysisException {
+Map> identityToValues = Maps.newHashMap();
+Map> identityToPartitionIds = Maps.newHashMap();
+MTMVPartitionExprService exprSerice =
MTMVPartitionExprFactory.getExprSerice(mvPartitionInfo.getExpr());
+
+for (Entry> entry :
relatedPartitionDescs.entrySet()) {
+String rollUpIdentity =
exprSerice.getRollUpIdentity(entry.getKey(), mvProperties);
Review Comment:
According to the existing logic, it won't be null. Are you worried about
supporting more expressions in the future?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) fuzzy spill and fix bugs [doris]
doris-robot commented on PR #33291: URL: https://github.com/apache/doris/pull/33291#issuecomment-2106573945 TPC-DS: Total hot run time: 186275 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit af927e77abedb25dcf04640f9585e1d56ad9eb23, data reload: false query1 900 360 353 353 query2 6540241524642415 query3 6680207 213 207 query4 23022 21254 21356 21254 query5 4169416 419 416 query6 267 171 173 171 query7 4585302 292 292 query8 243 189 188 188 query9 8556243924342434 query10 436 252 260 252 query11 14908 14450 14173 14173 query12 138 90 92 90 query13 1655393 384 384 query14 10293 680674456806 query15 208 174 169 169 query16 7689270 285 270 query17 1716564 546 546 query18 1964284 279 279 query19 194 156 156 156 query20 96 86 85 85 query21 204 132 171 132 query22 5091486548354835 query23 34340 33452 33480 33452 query24 10998 285829202858 query25 632 362 359 359 query26 1212155 153 153 query27 2468323 322 322 query28 6825207420672067 query29 873 641 637 637 query30 252 145 152 145 query31 975 759 746 746 query32 96 53 54 53 query33 744 268 238 238 query34 992 481 478 478 query35 826 686 670 670 query36 1101932 943 932 query37 120 63 67 63 query38 2868276827782768 query39 1624157815391539 query40 193 123 120 120 query41 41 38 37 37 query42 101 95 94 94 query43 598 580 522 522 query44 1234731 748 731 query45 265 248 243 243 query46 1067731 702 702 query47 1986186318811863 query48 381 307 311 307 query49 863 388 393 388 query50 776 382 396 382 query51 6770659165286528 query52 110 89 90 89 query53 360 283 286 283 query54 846 439 442 439 query55 73 72 72 72 query56 243 222 223 222 query57 1233113311491133 query58 210 193 197 193 query59 3427316331263126 query60 249 234 244 234 query61 112 89 90 89 query62 682 484 476 476 query63 311 290 289 289 query64 8644737273647364 query65 3104307331103073 query66 789 381 353 353 query67 15414 15007 14931 14931 query68 4503548 541 541 query69 493 312 302 302 query70 1171115911811159 query71 408 270 274 270 query72 7212253723482348 query73 714 329 327 327 query74 6468611860396039 query75 3323265625932593 query76 26671011964 964 query77 390 264 266 264 query78 10724 10109 10064 10064 query79 1805526 529 526 query80 1150453 441 441 query81 568 219 222 219 query82 958 90 94 90 query83 253 165 170 165 query84 242 87 84 84 query85 1190274 262 262 query86 393 320 321 320 query87 3319315530783078 query88 3193245224502450 query89 469 395 391 391 query90 1994192 194 192 query91 126 100 151 100 query92 56 54 54 54 query93 1457513 499 499 query94 1230181 189 181 query95 408 310 305 305 query96 584 283 273 273 query97 3181300330273003 query98 237 221 218 218 query99 1169883 895 883 Total cold run time: 278637 ms Total hot run time: 186275 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [Feature](Row store) support column group with store row format for partial columns of table [doris]
xiaokang commented on code in PR #34089:
URL: https://github.com/apache/doris/pull/34089#discussion_r1597773780
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -273,6 +273,8 @@ public class SessionVariable implements Serializable,
Writable {
public static final String ENABLE_PROJECTION = "enable_projection";
+public static final String ENABLE_SHORT_CIRCUIT_QUERY =
"enable_short_circuit_query";
Review Comment:
What's the purpose of this variable?
##
regression-test/suites/point_query_p0/test_rowstore.groovy:
##
@@ -16,6 +16,260 @@
// under the License.
suite("test_rowstore", "p0") {
+ // Parse url
+String jdbcUrl = context.config.jdbcUrl
+def user = context.config.jdbcUser
+def password = context.config.jdbcPassword
+String urlWithoutSchema = jdbcUrl.substring(jdbcUrl.indexOf("://") + 3)
+def sql_ip = urlWithoutSchema.substring(0, urlWithoutSchema.indexOf(":"))
+def realDb = "regression_test_point_query_p0"
+def sql_port
+if (urlWithoutSchema.indexOf("/") >= 0) {
+// e.g: jdbc:mysql://locahost:8080/?a=b
+sql_port = urlWithoutSchema.substring(urlWithoutSchema.indexOf(":") +
1, urlWithoutSchema.indexOf("/"))
+} else {
+// e.g: jdbc:mysql://locahost:8080
+sql_port = urlWithoutSchema.substring(urlWithoutSchema.indexOf(":") +
1)
+}
+def prepare_url = "jdbc:mysql://" + sql_ip + ":" + sql_port + "/" + realDb
+ "?=true"
+
+sql "DROP TABLE IF EXISTS table_with_column_group"
+sql """
+CREATE TABLE IF NOT EXISTS table_with_column_group (
+`k1` int(11) NULL COMMENT "",
+`v1` text NULL COMMENT "",
+`v2` bigint NULL COMMENT "",
+`v3` double NULL COMMENT "",
+`v4` datetime NULL COMMENT ""
+ ) ENGINE=OLAP
+ UNIQUE KEY(`k1`)
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "store_row_column" = "false",
+ "enable_unique_key_merge_on_write" = "true",
+ "light_schema_change" = "true",
+ "storage_format" = "V2",
+ "column_groups" = "group1:v1,v2,v3;group2:v2,v4"
+ )
+"""
+sql """
+insert into table_with_column_group values (1,
"11", 3, 4.0, '2021-02-01 11:11:11'), (2,
"2", 3, 4, '2022-02-01 11:11:11'), (3,
"", 3, 4, '2023-02-01 11:11:11');
+"""
+sql "set show_hidden_columns = true"
+qt_sql """
+select length(__DORIS_ROW_STORE_COL__group1),
length(__DORIS_ROW_STORE_COL__group2) from table_with_column_group order by k1;
+"""
+
+sql "DROP TABLE IF EXISTS table_with_column_group2"
+sql """
+CREATE TABLE IF NOT EXISTS table_with_column_group2 (
+`k1` int(11) NULL COMMENT "",
+`v1` text NULL COMMENT "",
+`v2` bigint NULL COMMENT "",
+`v3` double NULL COMMENT "",
+`v4` datetime NULL COMMENT ""
+ ) ENGINE=OLAP
+ UNIQUE KEY(`k1`)
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "store_row_column" = "false",
+ "enable_unique_key_merge_on_write" = "true",
+ "light_schema_change" = "true",
+ "storage_format" = "V2",
+ "column_groups" = "group2:v2"
+ )
+"""
+sql """
+insert into table_with_column_group2 values (1,
"11", 3, 4.0, '2021-02-01 11:11:11'), (2,
"2", 3, 4, '2022-02-01 11:11:11'), (3,
"", 3, 4, '2023-02-01 11:11:11');
+"""
+qt_sql """
+select length(__DORIS_ROW_STORE_COL__group2) from
table_with_column_group2 order by k1;
+"""
+
+sql "DROP TABLE IF EXISTS table_with_column_group3"
+sql """
+CREATE TABLE IF NOT EXISTS table_with_column_group3 (
+`k1` int(11) NULL COMMENT "",
+`v1` text NULL COMMENT "",
+`v2` bigint NULL COMMENT "",
+`v3` double NULL COMMENT "",
+`v4` datetime NULL COMMENT ""
+ ) ENGINE=OLAP
+ UNIQUE KEY(`k1`)
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "store_row_column" = "false",
+ "enable_unique_key_merge_on_write" = "true",
+ "light_schema_change" = "true",
+ "storage_format" = "V2",
+ "column_groups" = "group1:v2,v4"
+ )
+"""
+ sql """
+insert into
Re: [PR] [improvement](statistics)Use defaultSessionVariable instead of clone a new one. (#34672) [doris]
doris-robot commented on PR #34747: URL: https://github.com/apache/doris/pull/34747#issuecomment-2106571975 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 645f9e6ebeaf83dfb847b4ab1bad6c44b92db790 with default session variables Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 21.3 seconds inserted 1000 Rows, about 469K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] (planner) support auto aggregation for random distributed table [doris]
starocean999 commented on PR #33630: URL: https://github.com/apache/doris/pull/33630#issuecomment-2106571312 Generally, we can only fix the bug for nereids planner, unless there is a strong needs for branch 2.0.(the old planner will be discarded very soon). And we should consider not only MAX, MIN, SUM and also other agg types like HLL_UNION, BITMAP_UNION, QUANTILE_UNION and GENERIC(agg_state related) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](restore) restore failed between force_replication_allocation clusters [doris]
doris-robot commented on PR #34608: URL: https://github.com/apache/doris/pull/34608#issuecomment-2106571100 TPC-DS: Total hot run time: 186538 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ea194763fdf498554fc687f76b9ea8cc1f059123, data reload: false query1 905 362 346 346 query2 6463243724532437 query3 6657205 213 205 query4 24299 21173 21233 21173 query5 4192421 428 421 query6 273 187 184 184 query7 4588300 288 288 query8 233 188 191 188 query9 8694244024312431 query10 442 254 265 254 query11 14726 14210 14245 14210 query12 134 93 89 89 query13 1671373 362 362 query14 9810691778686917 query15 207 172 181 172 query16 7797260 261 260 query17 1872571 543 543 query18 1957273 264 264 query19 196 151 152 151 query20 95 84 84 84 query21 197 127 128 127 query22 5014483748994837 query23 34366 33570 33505 33505 query24 12085 300928722872 query25 666 373 371 371 query26 1883160 160 160 query27 3001324 334 324 query28 7510208420612061 query29 1168644 607 607 query30 304 157 154 154 query31 994 767 739 739 query32 92 53 55 53 query33 750 277 257 257 query34 1013490 491 490 query35 812 695 672 672 query36 1074938 937 937 query37 282 95 69 69 query38 2913278827512751 query39 1601157215611561 query40 273 132 127 127 query41 42 38 38 38 query42 107 95 96 95 query43 588 560 556 556 query44 1209723 745 723 query45 260 248 249 248 query46 1069722 737 722 query47 1943188318881883 query48 367 307 307 307 query49 1188389 406 389 query50 770 404 397 397 query51 6772667466496649 query52 106 90 94 90 query53 352 281 286 281 query54 949 440 459 440 query55 74 74 73 73 query56 243 224 225 224 query57 1238115811691158 query58 215 205 208 205 query59 3453305330313031 query60 270 243 267 243 query61 96 89 87 87 query62 673 476 475 475 query63 310 296 291 291 query64 9785738873937388 query65 3111309131203091 query66 1377352 334 334 query67 15480 15161 14883 14883 query68 4570531 543 531 query69 475 305 308 305 query70 1166114911701149 query71 407 278 278 278 query72 7259252823792379 query73 704 333 330 330 query74 6655615559515951 query75 3381263225892589 query76 28011000949 949 query77 452 273 264 264 query78 10631 10135 10091 10091 query79 2074510 524 510 query80 878 458 446 446 query81 508 223 230 223 query82 643 98 93 93 query83 241 163 177 163 query84 241 88 86 86 query85 1852289 321 289 query86 478 333 309 309 query87 3376311530873087 query88 4228247024752470 query89 477 387 407 387 query90 1995193 262 193 query91 130 98 100 98 query92 59 51 49 49 query93 2200521 499 499 query94 1260184 187 184 query95 404 309 309 309 query96 598 268 274 268 query97 3192305729892989 query98 237 220 228 220 query99 1103927 896 896 Total cold run time: 287931 ms Total hot run time: 186538 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [improvement](statistics)Use defaultSessionVariable instead of clone a new one. (#34672) [doris]
doris-robot commented on PR #34747: URL: https://github.com/apache/doris/pull/34747#issuecomment-2106569553 ClickBench: Total hot run time: 31.42 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 645f9e6ebeaf83dfb847b4ab1bad6c44b92db790, data reload: false query1 0.020.020.02 query2 0.060.020.02 query3 0.250.050.05 query4 1.780.070.06 query5 0.530.520.52 query6 1.230.660.62 query7 0.020.010.01 query8 0.030.030.02 query9 0.530.480.50 query10 0.540.540.53 query11 0.120.090.09 query12 0.120.090.08 query13 0.620.610.62 query14 0.780.790.78 query15 0.780.770.76 query16 0.350.390.38 query17 1.031.031.00 query18 0.230.260.23 query19 1.891.891.87 query20 0.010.010.01 query21 15.49 0.570.54 query22 2.142.121.70 query23 17.05 0.991.09 query24 5.911.551.44 query25 0.320.110.04 query26 0.820.140.15 query27 0.050.040.04 query28 6.450.720.72 query29 12.61 2.352.32 query30 0.620.570.57 query31 2.810.390.37 query32 3.380.500.49 query33 3.183.053.03 query34 15.24 4.794.80 query35 4.894.874.84 query36 1.061.011.01 query37 0.060.050.04 query38 0.040.020.02 query39 0.010.020.01 query40 0.160.140.14 query41 0.060.020.01 query42 0.020.010.01 query43 0.020.020.02 Total cold run time: 103.31 s Total hot run time: 31.42 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] (planner) support auto aggregation for random distributed table [doris]
starocean999 commented on code in PR #33630:
URL: https://github.com/apache/doris/pull/33630#discussion_r1597831263
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BuildAggForRandomDistributedTable.java:
##
@@ -0,0 +1,206 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.catalog.AggregateType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.DistributionInfo;
+import org.apache.doris.catalog.KeysType;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import
org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Min;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * build agg plan for querying random distributed table
+ */
+public class BuildAggForRandomDistributedTable implements AnalysisRuleFactory {
+
+@Override
+public List buildRules() {
+return ImmutableList.of(
+// Project(Scan)
+logicalProject(logicalOlapScan()).when(project ->
isRandomDistributedTbl(project.child()))
+.then(project -> preAggForRandomDistribution(project,
project.child()))
+
.toRule(RuleType.BUILD_AGG_FOR_RANDOM_DISTRIBUTED_TABLE_PROJECT_SCAN),
+// agg(scan)
+logicalAggregate(logicalOlapScan()).when(agg ->
isRandomDistributedTbl(agg.child())).whenNot(agg -> {
+Set functions =
agg.getAggregateFunctions();
+List groupByExprs =
agg.getGroupByExpressions();
+return functions.stream().allMatch(this::aggTypeMatch) &&
groupByExprs.stream()
+.allMatch(this::isKeyOrConstantExpr);
+})
+.then(agg -> preAggForRandomDistribution(agg,
agg.child()))
+
.toRule(RuleType.BUILD_AGG_FOR_RANDOM_DISTRIBUTED_TABLE_AGG_SCAN),
+// filter(scan)
+logicalFilter(logicalOlapScan()).when(filter ->
isRandomDistributedTbl(filter.child()))
+.then(filter -> preAggForRandomDistribution(filter,
filter.child()))
+
.toRule(RuleType.BUILD_AGG_FOR_RANDOM_DISTRIBUTED_TABLE_FILTER_SCAN));
+
+}
+
+/**
+ * check the olapTable of olapScan is randomDistributed table
+ *
+ * @param olapScan olap scan plan
+ * @return true if olapTable is randomDistributed table
+ */
+private boolean isRandomDistributedTbl(LogicalOlapScan olapScan) {
+OlapTable olapTable = olapScan.getTable();
+KeysType keysType = olapTable.getKeysType();
+DistributionInfo distributionInfo =
olapTable.getDefaultDistributionInfo();
+return keysType == KeysType.AGG_KEYS
+&& distributionInfo.getType() ==
DistributionInfo.DistributionInfoType.RANDOM;
+}
+
+/**
+ * add LogicalAggregate above olapScan for preAgg
+ *
+ * @param logicalPlan parent plan of olapScan
+ * @param olapScan olap scan plan, it may be LogicalProject,
LogicalFilter, LogicalAggregate
+ *
(doris) branch master updated (ab554bee2c4 -> 0a102afc478)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ab554bee2c4 [Feature](topn) BE adaptive choose whether push topn filter down to storage layer (#34713) add 0a102afc478 [Chore](topn) add TTopnFilterContext (#34718) No new revisions were added by this update. Summary of changes: gensrc/thrift/PaloInternalService.thrift | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](topn) add TTopnFilterContext [doris]
BiteThet merged PR #34718: URL: https://github.com/apache/doris/pull/34718 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](topn) BE adaptive choose whether push topn filter down to storage layer [doris]
BiteThet merged PR #34713: URL: https://github.com/apache/doris/pull/34713 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Feature](topn) BE adaptive choose whether push topn filter down to storage layer (#34713)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ab554bee2c4 [Feature](topn) BE adaptive choose whether push topn
filter down to storage layer (#34713)
ab554bee2c4 is described below
commit ab554bee2c43e0064dde17526ab5355ecdbb4510
Author: Pxl
AuthorDate: Mon May 13 11:29:50 2024 +0800
[Feature](topn) BE adaptive choose whether push topn filter down to storage
layer (#34713)
support judge topn filter push down
topn filter will push down to storage layer when 2 case:
filter target is key column
table data model is dup/merge on write
---
be/src/pipeline/exec/olap_scan_operator.h | 7 ++-
be/src/pipeline/exec/scan_operator.cpp| 9 ++---
be/src/pipeline/exec/scan_operator.h | 16 +---
be/src/runtime/runtime_predicate.h| 4 +++-
be/src/vec/exec/scan/new_olap_scanner.cpp | 15 ++-
5 files changed, 30 insertions(+), 21 deletions(-)
diff --git a/be/src/pipeline/exec/olap_scan_operator.h
b/be/src/pipeline/exec/olap_scan_operator.h
index 15dfd821772..daff2167f7f 100644
--- a/be/src/pipeline/exec/olap_scan_operator.h
+++ b/be/src/pipeline/exec/olap_scan_operator.h
@@ -80,7 +80,12 @@ private:
bool _storage_no_merge() override;
-bool _push_down_topn() override { return true; }
+bool _push_down_topn(const vectorized::RuntimePredicate& predicate)
override {
+if (!predicate.target_is_slot()) {
+return false;
+}
+return _is_key_column(predicate.get_col_name()) || _storage_no_merge();
+}
Status _init_scanners(std::list* scanners)
override;
diff --git a/be/src/pipeline/exec/scan_operator.cpp
b/be/src/pipeline/exec/scan_operator.cpp
index ddb379e0977..19a3911c6a7 100644
--- a/be/src/pipeline/exec/scan_operator.cpp
+++ b/be/src/pipeline/exec/scan_operator.cpp
@@ -189,9 +189,7 @@ Status
ScanLocalState::_normalize_conjuncts(RuntimeState* state) {
init_value_range(_slot_id_to_slot_desc[_colname_to_slot_id[colname]],
type);
}
-if (!_push_down_topn()) {
-RETURN_IF_ERROR(_get_topn_filters(state));
-}
+RETURN_IF_ERROR(_get_topn_filters(state));
for (auto it = _conjuncts.begin(); it != _conjuncts.end();) {
auto& conjunct = *it;
@@ -1269,11 +1267,8 @@ Status ScanLocalState::_init_profile() {
template
Status ScanLocalState::_get_topn_filters(RuntimeState* state) {
-for (auto id : get_topn_filter_source_node_ids()) {
+for (auto id : get_topn_filter_source_node_ids(state, false)) {
const auto& pred = state->get_query_ctx()->get_runtime_predicate(id);
-if (!pred.inited()) {
-continue;
-}
SlotDescriptor* slot_desc =
_slot_id_to_slot_desc[_colname_to_slot_id[pred.get_col_name()]];
vectorized::VExprSPtr topn_pred;
diff --git a/be/src/pipeline/exec/scan_operator.h
b/be/src/pipeline/exec/scan_operator.h
index 3ebd573fc71..e7ce9b31d19 100644
--- a/be/src/pipeline/exec/scan_operator.h
+++ b/be/src/pipeline/exec/scan_operator.h
@@ -158,8 +158,18 @@ class ScanLocalState : public ScanLocalStateBase {
std::vector dependencies() const override { return
{_scan_dependency.get()}; }
-std::vector get_topn_filter_source_node_ids() {
-return _parent->cast().topn_filter_source_node_ids;
+std::vector get_topn_filter_source_node_ids(RuntimeState* state, bool
push_down) {
+std::vector result;
+for (int id : _parent->cast().topn_filter_source_node_ids) {
+const auto& pred =
state->get_query_ctx()->get_runtime_predicate(id);
+if (!pred.inited()) {
+continue;
+}
+if (_push_down_topn(pred) == push_down) {
+result.push_back(id);
+}
+}
+return result;
}
protected:
@@ -176,7 +186,7 @@ protected:
virtual bool _should_push_down_common_expr() { return false; }
virtual bool _storage_no_merge() { return false; }
-virtual bool _push_down_topn() { return false; }
+virtual bool _push_down_topn(const vectorized::RuntimePredicate&
predicate) { return false; }
virtual bool _is_key_column(const std::string& col_name) { return false; }
virtual vectorized::VScanNode::PushDownType
_should_push_down_bloom_filter() {
return vectorized::VScanNode::PushDownType::UNACCEPTABLE;
diff --git a/be/src/runtime/runtime_predicate.h
b/be/src/runtime/runtime_predicate.h
index 00fbd62dd88..0305994e0fc 100644
--- a/be/src/runtime/runtime_predicate.h
+++ b/be/src/runtime/runtime_predicate.h
@@ -47,6 +47,7 @@ public:
Status init(PrimitiveType type, bool nulls_first, bool is_asc, const
std::string& col_name);
bool inited() const {
+// when sort node and scan node are not in the same fragment,
Re: [PR] [fix] (planner) support auto aggregation for random distributed table [doris]
starocean999 commented on code in PR #33630:
URL: https://github.com/apache/doris/pull/33630#discussion_r1597830385
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BuildAggForRandomDistributedTable.java:
##
@@ -0,0 +1,206 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.catalog.AggregateType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.DistributionInfo;
+import org.apache.doris.catalog.KeysType;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import
org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Min;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * build agg plan for querying random distributed table
+ */
+public class BuildAggForRandomDistributedTable implements AnalysisRuleFactory {
+
+@Override
+public List buildRules() {
+return ImmutableList.of(
+// Project(Scan)
+logicalProject(logicalOlapScan()).when(project ->
isRandomDistributedTbl(project.child()))
Review Comment:
better to simply this rule by only match logicalOlapScan with randomDist,
then always gererate a agg node above it. The MergeAggregate rule can merge
redundant agg later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) do not push rf through cteAnchor [doris]
morrySnow closed pull request #34744: [fix](nereids) do not push rf through cteAnchor URL: https://github.com/apache/doris/pull/34744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) do not push rf through cteAnchor [doris]
morrySnow commented on PR #34744: URL: https://github.com/apache/doris/pull/34744#issuecomment-2106566952 Using an alternative approach to fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)do not push rf through cteAnchor [doris]
morrySnow closed pull request #34742: [fix](nereids)do not push rf through cteAnchor URL: https://github.com/apache/doris/pull/34742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)do not push rf through cteAnchor [doris]
morrySnow commented on PR #34742: URL: https://github.com/apache/doris/pull/34742#issuecomment-2106566871 Using an alternative approach to fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)Use defaultSessionVariable instead of clone a new one. (#34672) [doris]
doris-robot commented on PR #34747: URL: https://github.com/apache/doris/pull/34747#issuecomment-2106565922 TPC-DS: Total hot run time: 202907 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 645f9e6ebeaf83dfb847b4ab1bad6c44b92db790, data reload: false query1 928 388 376 376 query2 6514287528102810 query3 6921216 208 208 query4 20171 18014 17908 17908 query5 19747 655166016551 query6 282 216 239 216 query7 4204304 315 304 query8 265 228 266 228 query9 3158274026642664 query10 424 285 286 285 query11 11199 10574 10725 10574 query12 128 74 71 71 query13 5590706 684 684 query14 18210 13353 13624 13353 query15 367 224 227 224 query16 6450273 262 262 query17 15321486867 867 query18 2275407 410 407 query19 207 149 153 149 query20 82 75 82 75 query21 192 92 93 92 query22 5280508150055005 query23 32590 32005 32088 32005 query24 6919650265566502 query25 510 423 420 420 query26 526 166 164 164 query27 1883301 299 299 query28 6161237223402340 query29 2872263828122638 query30 239 172 164 164 query31 910 751 756 751 query32 71 64 57 57 query33 388 275 258 258 query34 853 460 480 460 query35 1114965 859 859 query36 1351156611121112 query37 92 56 61 56 query38 3067292629632926 query39 1377132913201320 query40 201 89 97 89 query41 39 37 37 37 query42 80 91 92 91 query43 743 802 749 749 query44 1131715 724 715 query45 243 228 232 228 query46 1227967 988 967 query47 1819179618191796 query48 1007732 731 731 query49 608 385 376 376 query50 851 621 580 580 query51 4782467546304630 query52 86 79 77 77 query53 451 319 328 319 query54 2652244024932440 query55 80 84 78 78 query56 230 204 196 196 query57 1135114812411148 query58 212 202 211 202 query59 4264402639003900 query60 211 205 208 205 query61 93 82 85 82 query62 802 516 489 489 query63 488 348 340 340 query64 2341151114341434 query65 3650362435893589 query66 800 368 380 368 query67 16164 15030 15176 15030 query68 8514628 635 628 query69 562 343 351 343 query70 1747140214481402 query71 405 325 318 318 query72 6427337833803378 query73 731 338 323 323 query74 6250584358605843 query75 4996372137123712 query76 5098117110741074 query77 713 259 247 247 query78 12514 11486 11724 11486 query79 8883638 650 638 query80 2019389 389 389 query81 501 233 228 228 query82 1680101 98 98 query83 198 133 132 132 query84 254 67 70 67 query85 1001308 293 293 query86 350 285 333 285 query87 3194302330463023 query88 5276236123712361 query89 385 291 300 291 query90 1883224 219 219 query91 182 133 135 133 query92 59 52 52 52 query93 5845615 550 550 query94 744 205 210 205 query95 1101105110471047 query96 650 337 324 324 query97 6477639162846284 query98 193 186 173 173 query99 2942831 919 831 Total cold run time: 313213 ms Total hot run time: 202907 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the problem that DLF cannot be used normally [doris]
CalvinKirs commented on PR #34749: URL: https://github.com/apache/doris/pull/34749#issuecomment-2106565768 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](prepare) fix datetimev2 return err when binary_row_format [doris]
github-actions[bot] commented on PR #34662: URL: https://github.com/apache/doris/pull/34662#issuecomment-2106565772 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) fuzzy spill and fix bugs [doris]
doris-robot commented on PR #33291: URL: https://github.com/apache/doris/pull/33291#issuecomment-2106565698 TPC-H: Total hot run time: 41792 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit af927e77abedb25dcf04640f9585e1d56ad9eb23, data reload: false -- Round 1 -- q1 17609 433242164216 q2 2021208 200 200 q3 10459 122912111211 q4 10213 795 859 795 q5 7456269526802680 q6 219 135 132 132 q7 1022596 582 582 q8 9227212220882088 q9 9431671667086708 q10 9221377238433772 q11 448 244 249 244 q12 475 233 232 232 q13 17280 318033073180 q14 269 237 215 215 q15 522 470 488 470 q16 494 414 411 411 q17 977 743 678 678 q18 8350778277017701 q19 4434149415431494 q20 622 326 314 314 q21 5284427141884188 q22 337 286 281 281 Total cold run time: 116370 ms Total hot run time: 41792 ms - Round 2, with runtime_filter_mode=off - q1 4562442344904423 q2 373 283 280 280 q3 3250304728952895 q4 1994166916821669 q5 5504548655185486 q6 215 123 122 122 q7 2325201719461946 q8 3251337733823377 q9 8694864085758575 q10 3978382238443822 q11 586 517 498 498 q12 782 649 630 630 q13 15876 327031903190 q14 313 276 285 276 q15 522 492 501 492 q16 489 410 421 410 q17 1762148314801480 q18 7615755373147314 q19 1722156915761569 q20 2003175117571751 q21 5107497047184718 q22 571 481 492 481 Total cold run time: 71494 ms Total hot run time: 55404 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](topn) add TTopnFilterContext [doris]
github-actions[bot] commented on PR #34718: URL: https://github.com/apache/doris/pull/34718#issuecomment-2106565576 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](IO) Replace file writer's finalize function with async close [doris]
doris-robot commented on PR #34679: URL: https://github.com/apache/doris/pull/34679#issuecomment-2106565173 TPC-DS: Total hot run time: 185963 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 15a8aca2ef26f033977acef1f6e73195a76fda1a, data reload: false query1 917 364 348 348 query2 7161227422502250 query3 6650211 207 207 query4 22874 21141 21168 21141 query5 4141418 414 414 query6 254 189 176 176 query7 4576301 288 288 query8 255 197 189 189 query9 8547241324102410 query10 430 251 262 251 query11 14777 14204 14128 14128 query12 140 92 91 91 query13 1651383 368 368 query14 10204 683283296832 query15 259 179 179 179 query16 8070282 277 277 query17 1696580 568 568 query18 2108284 281 281 query19 205 160 157 157 query20 94 87 88 87 query21 196 126 133 126 query22 5019482348964823 query23 34258 33397 33442 33397 query24 10611 287029272870 query25 558 365 361 361 query26 694 156 149 149 query27 2190320 315 315 query28 5771207920462046 query29 827 635 588 588 query30 252 151 151 151 query31 996 780 728 728 query32 94 52 56 52 query33 642 250 240 240 query34 864 485 481 481 query35 823 678 659 659 query36 1079921 904 904 query37 104 69 67 67 query38 2885274827732748 query39 1592156515671565 query40 197 125 123 123 query41 42 40 38 38 query42 106 97 96 96 query43 600 536 542 536 query44 1098724 735 724 query45 272 221 256 221 query46 1084725 711 711 query47 1945188218941882 query48 375 301 290 290 query49 878 394 383 383 query50 749 386 385 385 query51 6848681667766776 query52 104 87 89 87 query53 344 281 283 281 query54 856 445 433 433 query55 78 77 78 77 query56 251 219 223 219 query57 1240115111451145 query58 210 197 202 197 query59 3347326930713071 query60 258 232 236 232 query61 92 90 88 88 query62 648 482 462 462 query63 309 284 290 284 query64 8497736772997299 query65 3153307930893079 query66 775 342 342 342 query67 15284 15191 15168 15168 query68 4553533 572 533 query69 488 300 302 300 query70 11891148 query71 381 272 273 272 query72 7361260923292329 query73 701 330 331 330 query74 6476610260436043 query75 3344265225992599 query76 2435945 1009945 query77 566 266 270 266 query78 10600 10156 10094 10094 query79 2357532 520 520 query80 957 431 438 431 query81 526 219 228 219 query82 952 98 91 91 query83 238 163 167 163 query84 249 86 84 84 query85 1166318 263 263 query86 434 334 316 316 query87 3256315331373137 query88 4057233823372337 query89 466 379 370 370 query90 1966188 188 188 query91 129 106 97 97 query92 58 49 47 47 query93 1872528 507 507 query94 1243180 184 180 query95 405 299 312 299 query96 589 277 266 266 query97 3169297029692969 query98 243 216 214 214 query99 1243902 914 902 Total cold run time: 23 ms Total hot run time: 185963 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [Chore](topn) add TTopnFilterContext [doris]
github-actions[bot] commented on PR #34718: URL: https://github.com/apache/doris/pull/34718#issuecomment-2106564970 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](topn) BE adaptive choose whether push topn filter down to storage layer [doris]
github-actions[bot] commented on PR #34713: URL: https://github.com/apache/doris/pull/34713#issuecomment-2106564103 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](prepare) fix datetimev2 return err when binary_row_format [doris]
github-actions[bot] commented on PR #34662: URL: https://github.com/apache/doris/pull/34662#issuecomment-2106564018 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](user manager) Add rwlock for userManager [doris]
deardeng commented on PR #34699: URL: https://github.com/apache/doris/pull/34699#issuecomment-2106563282 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the problem that DLF cannot be used normally [doris]
doris-robot commented on PR #34749: URL: https://github.com/apache/doris/pull/34749#issuecomment-2106563263 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
