Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
github-actions[bot] closed pull request #48866: [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method URL: https://github.com/apache/doris/pull/48866 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
github-actions[bot] commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-3336308614 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
Tech-Circle-48 commented on code in PR #48866:
URL: https://github.com/apache/doris/pull/48866#discussion_r2019711150
##
be/src/olap/delta_writer.cpp:
##
@@ -231,7 +231,24 @@ int64_t BaseDeltaWriter::mem_consumption(MemType mem) {
return _memtable_writer->mem_consumption(mem);
}
-void DeltaWriter::_request_slave_tablet_pull_rowset(const PNodeInfo&
node_info) {
+Status DeltaWriter::_safe_get_file_size(const std::string& file_path, int64_t*
file_size) {
+CHECK(file_size != nullptr) << "Null output parameter in
safe_get_file_size";
+
+try {
+if (!std::filesystem::exists(file_path)) {
Review Comment:
the issue has been fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
github-actions[bot] commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2763357914 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
hello-stephen commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2739152357 # BE UT Coverage Report Increment line coverage `0.00% (0/24)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/0337dd5735d7b17e406da0e91a287c7f7cb1b991_0337dd5735d7b17e406da0e91a287c7f7cb1b991/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/0337dd5735d7b17e406da0e91a287c7f7cb1b991_0337dd5735d7b17e406da0e91a287c7f7cb1b991/report/index.html) | Category | Coverage | |---|| | Function Coverage | 48.79% (13066/26782) | | Line Coverage | 38.37% (112690/293684) | | Region Coverage | 37.16% (57280/154142) | | Branch Coverage | 32.27% (28793/89236) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2739034058 ClickBench: Total hot run time: 30.82 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 0337dd5735d7b17e406da0e91a287c7f7cb1b991, data reload: false query1 0.040.040.03 query2 0.120.110.10 query3 0.260.200.19 query4 1.590.190.20 query5 0.580.580.57 query6 1.160.710.72 query7 0.020.020.02 query8 0.040.030.03 query9 0.580.510.51 query10 0.580.590.57 query11 0.150.110.10 query12 0.160.120.12 query13 0.610.600.60 query14 2.672.692.82 query15 0.940.850.85 query16 0.380.360.38 query17 1.011.051.03 query18 0.210.200.20 query19 1.861.961.84 query20 0.010.010.01 query21 15.35 0.870.55 query22 0.761.210.77 query23 14.75 1.400.62 query24 7.131.160.45 query25 0.470.270.09 query26 0.500.150.14 query27 0.050.050.06 query28 10.29 0.940.43 query29 12.53 3.943.29 query30 0.260.090.06 query31 2.830.610.39 query32 3.230.550.46 query33 3.002.983.08 query34 15.89 5.064.49 query35 4.504.554.52 query36 0.690.480.48 query37 0.090.060.06 query38 0.050.040.04 query39 0.030.020.02 query40 0.170.140.13 query41 0.090.030.03 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 105.7 s Total hot run time: 30.82 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2739029232 TPC-DS: Total hot run time: 192548 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0337dd5735d7b17e406da0e91a287c7f7cb1b991, data reload: false query1 1396106310361036 query2 6094196519391939 query3 11069 460646004600 query4 56027 25729 23076 23076 query5 5029581 485 485 query6 349 197 183 183 query7 4906503 288 288 query8 311 248 233 233 query9 5778259426032594 query10 415 301 246 246 query11 15093 15328 15010 15010 query12 166 112 107 107 query13 1068491 383 383 query14 10347 684468796844 query15 214 201 187 187 query16 7084656 516 516 query17 1103768 578 578 query18 1550441 333 333 query19 203 202 181 181 query20 134 128 118 118 query21 211 132 112 112 query22 4318444843574357 query23 33966 33411 33275 33275 query24 5957244524862445 query25 475 464 434 434 query26 678 276 158 158 query27 1897504 331 331 query28 2883246224502450 query29 566 578 420 420 query30 272 222 189 189 query31 877 863 783 783 query32 67 63 58 58 query33 465 389 295 295 query34 762 838 510 510 query35 816 845 728 728 query36 974 1003907 907 query37 135 105 75 75 query38 4247426441464146 query39 1499159714201420 query40 203 119 105 105 query41 54 51 49 49 query42 129 109 101 101 query43 499 519 497 497 query44 1330813 809 809 query45 181 172 164 164 query46 844 1052651 651 query47 1823188018571857 query48 385 424 325 325 query49 722 521 424 424 query50 714 775 439 439 query51 4224433642794279 query52 110 108 99 99 query53 235 264 197 197 query54 494 521 422 422 query55 92 84 85 84 query56 300 261 272 261 query57 1160120411031103 query58 252 254 250 250 query59 2843294628252825 query60 290 273 262 262 query61 120 118 117 117 query62 744 729 673 673 query63 226 216 189 189 query64 15171024704 704 query65 4633450843824382 query66 708 394 302 302 query67 15788 15640 15424 15424 query68 7426814 500 500 query69 546 316 251 251 query70 1227111211071107 query71 540 291 270 270 query72 5537361037593610 query73 1445741 345 345 query74 9039905987458745 query75 3795318627142714 query76 42541176749 749 query77 666 364 282 282 query78 10119 10051 92329232 query79 2720838 593 593 query80 694 517 448 448 query81 485 265 220 220 query82 680 125 95 95 query83 275 174 150 150 query84 287 89 75 75 query85 775 342 309 309 query86 388 288 304 288 query87 4422450843894389 query88 3592229022242224 query89 422 310 289 289 query90 1844211 204 204 query91 139 141 107 107 query92 71 58 58 58 query93 19901051574 574 query94 669 405 287 287 query95 344 271 259 259 query96 487 556 351 351 query97 346933253325 query98 233 207 217 207 query99 1468141512671267 Total cold run time: 299933 ms Total hot run time: 192548 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2739014512 TPC-H: Total hot run time: 32645 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0337dd5735d7b17e406da0e91a287c7f7cb1b991, data reload: false -- Round 1 -- q1 24150 508950155015 q2 2062297 204 204 q3 10351 1255683 683 q4 10215 1002572 572 q5 7526232623442326 q6 191 169 140 140 q7 912 747 612 612 q8 9321128811121112 q9 4846483949064839 q10 6822232618891889 q11 484 273 257 257 q12 351 353 221 221 q13 17789 367730923092 q14 243 239 220 220 q15 530 468 472 468 q16 617 613 590 590 q17 579 856 349 349 q18 6891648663176317 q19 1221960 565 565 q20 338 337 216 216 q21 3218229019711971 q22 10471081987 987 Total cold run time: 109704 ms Total hot run time: 32645 ms - Round 2, with runtime_filter_mode=off - q1 5131509251165092 q2 240 328 235 235 q3 2183269423422342 q4 1449188113801380 q5 4305420844614208 q6 215 168 131 131 q7 1997198318071807 q8 2597263424832483 q9 7260725372047204 q10 2996318926612661 q11 571 524 512 512 q12 663 798 626 626 q13 3493386033343334 q14 295 306 266 266 q15 523 466 473 466 q16 624 695 650 650 q17 1140161413131313 q18 7774773774707470 q19 836 841 877 841 q20 1964203919191919 q21 5315494448464846 q22 1114103810211021 Total cold run time: 52685 ms Total hot run time: 50807 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
morningman commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2738962859 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
liaoxin01 commented on code in PR #48866:
URL: https://github.com/apache/doris/pull/48866#discussion_r1997621894
##
be/src/olap/delta_writer.cpp:
##
@@ -231,7 +231,24 @@ int64_t BaseDeltaWriter::mem_consumption(MemType mem) {
return _memtable_writer->mem_consumption(mem);
}
-void DeltaWriter::_request_slave_tablet_pull_rowset(const PNodeInfo&
node_info) {
+Status DeltaWriter::_safe_get_file_size(const std::string& file_path, int64_t*
file_size) {
+CHECK(file_size != nullptr) << "Null output parameter in
safe_get_file_size";
+
+try {
+if (!std::filesystem::exists(file_path)) {
Review Comment:
There is no need to check if the file exists. If the file does not exist,
the file_size function will throw an exception.
##
be/src/olap/delta_writer.cpp:
##
@@ -281,7 +299,9 @@ void DeltaWriter::_request_slave_tablet_pull_rowset(const
PNodeInfo& node_info)
for (int segment_id = 0; segment_id <
cur_rowset->rowset_meta()->num_segments(); segment_id++) {
auto seg_path =
local_segment_path(tablet_path,
cur_rowset->rowset_id().to_string(), segment_id);
-int64_t segment_size = std::filesystem::file_size(seg_path);
+int64_t segment_size = 0;
+RETURN_IF_ERROR(safe_get_file_size(seg_path, &segment_size));
Review Comment:
```suggestion
RETURN_IF_ERROR(_safe_get_file_size(seg_path, &segment_size));
```
compile error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
hello-stephen commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2710577655 # BE UT Coverage Report Increment line coverage `0.00% (0/42)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/9895c431a8a55d9f0c9edf120705e65b2de79734_9895c431a8a55d9f0c9edf120705e65b2de79734/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/9895c431a8a55d9f0c9edf120705e65b2de79734_9895c431a8a55d9f0c9edf120705e65b2de79734/report/index.html) | Category | Coverage | |---|| | Function Coverage | 46.83% (12519/26733) | | Line Coverage | 36.44% (106688/292778) | | Region Coverage | 35.47% (54510/153662) | | Branch Coverage | 30.85% (27424/2) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
github-actions[bot] commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2724251812 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2710357349 TPC-H: Total hot run time: 32752 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9895c431a8a55d9f0c9edf120705e65b2de79734, data reload: false -- Round 1 -- q1 17644 531251485148 q2 2044292 162 162 q3 10422 1324698 698 q4 10222 1001536 536 q5 7521242023192319 q6 192 164 133 133 q7 912 745 595 595 q8 9309125811831183 q9 4971482148804821 q10 6845230418901890 q11 474 283 258 258 q12 356 362 229 229 q13 17778 372031133113 q14 233 225 212 212 q15 523 486 471 471 q16 625 616 569 569 q17 557 864 337 337 q18 6998636863856368 q19 1203929 571 571 q20 333 328 204 204 q21 2942218919431943 q22 10781019992 992 Total cold run time: 103182 ms Total hot run time: 32752 ms - Round 2, with runtime_filter_mode=off - q1 5273522052085208 q2 245 326 232 232 q3 2164267323152315 q4 1471182513611361 q5 4261412941604129 q6 203 160 123 123 q7 1852199617701770 q8 2631269125982598 q9 7203720570897089 q10 2980322027612761 q11 573 509 495 495 q12 699 753 613 613 q13 3521397932943294 q14 282 315 271 271 q15 517 485 483 483 q16 655 704 644 644 q17 1157162013531353 q18 7740763675547554 q19 898 885 1056885 q20 1923206519001900 q21 5465500049874987 q22 10681027997 997 Total cold run time: 52781 ms Total hot run time: 51062 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
sollhui commented on code in PR #48866:
URL: https://github.com/apache/doris/pull/48866#discussion_r1988763963
##
be/src/olap/delta_writer.cpp:
##
@@ -231,7 +239,26 @@ int64_t BaseDeltaWriter::mem_consumption(MemType mem) {
return _memtable_writer->mem_consumption(mem);
}
-void DeltaWriter::_request_slave_tablet_pull_rowset(const PNodeInfo&
node_info) {
+Status safe_get_file_size(const std::string& file_path, int64_t* file_size) {
Review Comment:
Can you simplify it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
morningman commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2724251281 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
github-actions[bot] commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2724219554 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
github-actions[bot] commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2724219486 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
Tech-Circle-48 commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2724205809 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
hello-stephen commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2710152205 # BE UT Coverage Report Increment line coverage `0.00% (0/29)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/4005ab59daead76c5b48bc4319cb52d6edb9fbe1_4005ab59daead76c5b48bc4319cb52d6edb9fbe1/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/4005ab59daead76c5b48bc4319cb52d6edb9fbe1_4005ab59daead76c5b48bc4319cb52d6edb9fbe1/report/index.html) | Category | Coverage | |---|| | Function Coverage | 46.83% (12519/26733) | | Line Coverage | 36.45% (106699/292754) | | Region Coverage | 35.48% (54520/153646) | | Branch Coverage | 30.86% (27429/88868) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
morningman commented on code in PR #48866:
URL: https://github.com/apache/doris/pull/48866#discussion_r1991776701
##
be/src/olap/delta_writer.h:
##
@@ -144,7 +144,9 @@ class DeltaWriter final : public BaseDeltaWriter {
private:
void _init_profile(RuntimeProfile* profile) override;
-void _request_slave_tablet_pull_rowset(const PNodeInfo& node_info);
+int64_t safe_get_filesize(const std::string& file_path);
Review Comment:
```suggestion
int64_t _safe_get_filesize(const std::string& file_path);
```
##
be/src/olap/delta_writer.cpp:
##
@@ -231,7 +231,26 @@ int64_t BaseDeltaWriter::mem_consumption(MemType mem) {
return _memtable_writer->mem_consumption(mem);
}
-void DeltaWriter::_request_slave_tablet_pull_rowset(const PNodeInfo&
node_info) {
+Status safe_get_file_size(const std::string& file_path, int64_t* file_size) {
+if (file_size == nullptr) {
+return Status::InvalidArgument("Null output parameter in
safe_get_file_size");
Review Comment:
I think you can just use `CHECK(file_size != nullptr) << "Null output
parameter in safe_get_file_size"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2710387983 TPC-DS: Total hot run time: 186036 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9895c431a8a55d9f0c9edf120705e65b2de79734, data reload: false query1 1022387 400 387 query2 6561192719571927 query3 6790215 221 215 query4 26425 23496 23570 23496 query5 4699672 512 512 query6 318 201 188 188 query7 4604498 317 317 query8 295 244 234 234 query9 8624266126612661 query10 468 307 256 256 query11 15884 15242 14897 14897 query12 168 110 106 106 query13 1651519 406 406 query14 11225 688465896589 query15 261 204 170 170 query16 7555637 431 431 query17 1564735 560 560 query18 1907402 319 319 query19 238 192 160 160 query20 121 120 118 118 query21 271 119 103 103 query22 4272421642034203 query23 33885 32838 32816 32816 query24 6972240124412401 query25 531 443 387 387 query26 1221272 152 152 query27 1995480 329 329 query28 3906242424332424 query29 666 562 431 431 query30 282 215 190 190 query31 956 882 812 812 query32 74 65 64 64 query33 550 354 307 307 query34 800 838 501 501 query35 810 818 738 738 query36 962 995 899 899 query37 132 133 78 78 query38 4173411941424119 query39 1452138613871386 query40 201 120 102 102 query41 54 60 61 60 query42 119 106 103 103 query43 523 520 480 480 query44 1344808 800 800 query45 181 172 166 166 query46 872 999 632 632 query47 1733177617001700 query48 382 414 302 302 query49 785 550 436 436 query50 726 736 408 408 query51 4233413941544139 query52 104 112 95 95 query53 243 270 188 188 query54 499 491 418 418 query55 82 80 86 80 query56 268 272 262 262 query57 1120113510681068 query58 248 237 240 237 query59 2816270426142614 query60 282 270 281 270 query61 121 120 117 117 query62 797 740 662 662 query63 233 194 189 189 query64 42021057670 670 query65 4451432743634327 query66 1046402 300 300 query67 15756 16010 15443 15443 query68 7798895 510 510 query69 464 294 268 268 query70 1226115710871087 query71 461 290 272 272 query72 5617369538183695 query73 818 741 355 355 query74 8935918786788678 query75 3876313526962696 query76 36901255756 756 query77 809 389 292 292 query78 10022 10230 93329332 query79 3133834 583 583 query80 598 509 437 437 query81 482 261 229 229 query82 726 124 93 93 query83 177 167 154 154 query84 239 92 75 75 query85 779 430 313 313 query86 383 311 292 292 query87 4561474843564356 query88 3635220822202208 query89 397 310 287 287 query90 1839208 211 208 query91 136 137 106 106 query92 76 60 59 59 query93 18161048582 582 query94 671 399 304 304 query95 357 262 261 261 query96 481 555 277 277 query97 3344336532933293 query98 228 208 199 199 query99 1328140812221222 Total cold run time: 276813 ms Total hot run time: 186036 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2716416649 TPC-DS: Total hot run time: 185250 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 66537c74e9b00ed7930a1f7346b41ab40c9784b7, data reload: false query1 1011376 379 376 query2 6530189118891889 query3 6792218 216 216 query4 26042 23231 23395 23231 query5 4362624 473 473 query6 296 204 197 197 query7 4599498 300 300 query8 308 250 242 242 query9 8611260925872587 query10 475 304 254 254 query11 15548 15170 14823 14823 query12 162 110 107 107 query13 1654524 407 407 query14 8844626064506260 query15 205 186 175 175 query16 7108611 482 482 query17 1206706 564 564 query18 1802405 315 315 query19 189 187 164 164 query20 126 117 115 115 query21 220 125 110 110 query22 4374430341964196 query23 33777 33065 33232 33065 query24 8287241623722372 query25 532 459 374 374 query26 1240272 154 154 query27 2694492 331 331 query28 4346241624002400 query29 732 569 433 433 query30 290 213 192 192 query31 937 838 768 768 query32 71 63 62 62 query33 558 360 304 304 query34 768 844 483 483 query35 798 796 764 764 query36 956 993 874 874 query37 119 95 73 73 query38 4166406640954066 query39 1460140413891389 query40 217 115 104 104 query41 55 74 51 51 query42 114 101 102 101 query43 480 505 486 486 query44 1264791 767 767 query45 177 169 163 163 query46 827 1030625 625 query47 1732179817381738 query48 376 403 299 299 query49 791 497 436 436 query50 690 717 391 391 query51 4129420141184118 query52 109 110 95 95 query53 229 263 178 178 query54 479 493 405 405 query55 88 82 85 82 query56 262 253 274 253 query57 1128116110741074 query58 242 238 240 238 query59 2539254025942540 query60 277 268 257 257 query61 125 122 128 122 query62 798 735 651 651 query63 231 186 182 182 query64 4416995 668 668 query65 4448430143894301 query66 1150401 292 292 query67 15767 15586 15334 15334 query68 8006873 493 493 query69 465 302 267 267 query70 1233114611171117 query71 469 287 256 256 query72 5668359038043590 query73 779 729 343 343 query74 9040937789278927 query75 3887316026882688 query76 37421172762 762 query77 780 377 277 277 query78 10068 10103 92889288 query79 2612811 589 589 query80 630 514 443 443 query81 495 259 222 222 query82 512 124 97 97 query83 209 167 147 147 query84 290 98 71 71 query85 779 426 300 300 query86 383 317 277 277 query87 4442440944114409 query88 3519223722402237 query89 385 322 287 287 query90 1875206 213 206 query91 143 141 106 106 query92 82 65 57 57 query93 18111044561 561 query94 669 420 313 313 query95 353 264 302 264 query96 482 560 273 273 query97 3301339533173317 query98 223 201 202 201 query99 1428141013271327 Total cold run time: 274095 ms Total hot run time: 185250 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2716422326 ClickBench: Total hot run time: 30.97 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 66537c74e9b00ed7930a1f7346b41ab40c9784b7, data reload: false query1 0.040.040.03 query2 0.070.040.03 query3 0.230.060.06 query4 1.650.100.10 query5 0.560.550.55 query6 1.210.720.71 query7 0.030.010.02 query8 0.050.040.03 query9 0.580.530.52 query10 0.570.600.58 query11 0.160.110.10 query12 0.140.110.11 query13 0.620.600.60 query14 2.792.692.79 query15 0.910.840.84 query16 0.390.380.38 query17 1.021.061.00 query18 0.210.190.19 query19 1.891.971.72 query20 0.020.010.01 query21 15.36 0.890.54 query22 0.751.150.57 query23 15.10 1.350.66 query24 6.841.911.23 query25 0.480.260.13 query26 0.640.150.13 query27 0.050.050.06 query28 9.480.780.44 query29 12.54 3.953.28 query30 0.260.090.06 query31 2.830.580.38 query32 3.240.540.45 query33 2.973.033.02 query34 15.81 5.094.46 query35 4.484.524.52 query36 0.660.500.48 query37 0.090.060.06 query38 0.050.040.04 query39 0.030.030.02 query40 0.170.130.12 query41 0.080.020.02 query42 0.040.030.02 query43 0.040.040.03 Total cold run time: 105.13 s Total hot run time: 30.97 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2716398172 TPC-H: Total hot run time: 32432 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 66537c74e9b00ed7930a1f7346b41ab40c9784b7, data reload: false -- Round 1 -- q1 17621 539850645064 q2 2035298 162 162 q3 10421 1363703 703 q4 10220 1048505 505 q5 7570239523632363 q6 192 166 131 131 q7 924 758 603 603 q8 9298127611271127 q9 4969469847614698 q10 6820232218881888 q11 458 270 256 256 q12 348 353 208 208 q13 17760 367530743074 q14 232 242 209 209 q15 530 482 482 482 q16 625 627 597 597 q17 582 889 339 339 q18 6866636663926366 q19 1498967 578 578 q20 313 327 186 186 q21 2798212318931893 q22 1036100510001000 Total cold run time: 103116 ms Total hot run time: 32432 ms - Round 2, with runtime_filter_mode=off - q1 5196515651235123 q2 239 326 219 219 q3 2170269322732273 q4 1440188313871387 q5 4216412041894120 q6 210 166 124 124 q7 1868195218391839 q8 2657275926272627 q9 7373720471607160 q10 2990322327812781 q11 581 499 503 499 q12 696 776 615 615 q13 3477387932213221 q14 288 301 268 268 q15 532 466 461 461 q16 639 674 638 638 q17 1136160113691369 q18 7754772575607560 q19 826 795 857 795 q20 1931203218921892 q21 5334498949544954 q22 10611016981 981 Total cold run time: 52614 ms Total hot run time: 50906 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
Tech-Circle-48 commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2716300283 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
sollhui commented on code in PR #48866: URL: https://github.com/apache/doris/pull/48866#discussion_r1986944568 ## be/src/olap/delta_writer.cpp: ## @@ -291,17 +305,19 @@ void DeltaWriter::_request_slave_tablet_pull_rowset(const PNodeInfo& node_info) std::string inverted_index_file = InvertedIndexDescriptor::get_index_file_path_v1( index_path_prefix, index_meta.first, index_meta.second); -int64_t size = std::filesystem::file_size(inverted_index_file); -PTabletWriteSlaveRequest::IndexSize index_size; -index_size.set_indexid(index_meta.first); -index_size.set_size(size); -index_size.set_suffix_path(index_meta.second); -// Fetch the map value for the current segment_id. -// If it doesn't exist, this will insert a new default-constructed IndexSizeMapValue -auto& index_size_map_value = - (*(request->mutable_inverted_indices_size()))[segment_id]; -// Add the new index size to the map value. -*index_size_map_value.mutable_index_sizes()->Add() = std::move(index_size); +int64_t size = safe_get_file_size(inverted_index_file); Review Comment: Should return directly instead of not executing the logic if there is an exception? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
hello-stephen commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2709895162 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
Tech-Circle-48 commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2709898131 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2709989158 TPC-DS: Total hot run time: 185736 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 4005ab59daead76c5b48bc4319cb52d6edb9fbe1, data reload: false query1 987 393 374 374 query2 6544193619461936 query3 6788213 219 213 query4 26657 23579 23434 23434 query5 4365689 493 493 query6 312 204 195 195 query7 4602496 308 308 query8 296 261 246 246 query9 8601262826322628 query10 498 312 268 268 query11 15716 15125 14846 14846 query12 157 111 111 111 query13 1666540 402 402 query14 8882626067486260 query15 221 195 176 176 query16 7522657 522 522 query17 1205695 543 543 query18 1962404 300 300 query19 189 179 162 162 query20 121 113 114 113 query21 242 123 101 101 query22 4180418542514185 query23 34077 33194 32935 32935 query24 7640239423692369 query25 526 447 389 389 query26 1217271 153 153 query27 2526462 337 337 query28 4268243624412436 query29 750 558 434 434 query30 308 213 188 188 query31 948 862 772 772 query32 73 60 64 60 query33 554 354 305 305 query34 806 820 519 519 query35 783 820 738 738 query36 964 984 871 871 query37 117 95 79 79 query38 4217417541844175 query39 1461140813941394 query40 207 113 101 101 query41 53 50 51 50 query42 116 102 105 102 query43 499 509 488 488 query44 1276796 786 786 query45 174 170 164 164 query46 826 1025615 615 query47 1783180717101710 query48 386 428 303 303 query49 791 506 426 426 query50 688 740 418 418 query51 4223417241664166 query52 110 104 100 100 query53 231 262 190 190 query54 497 500 406 406 query55 81 81 80 80 query56 287 264 280 264 query57 1112115810941094 query58 246 230 241 230 query59 2558284726002600 query60 299 281 250 250 query61 124 124 118 118 query62 816 740 666 666 query63 230 190 191 190 query64 43321021672 672 query65 4418433443944334 query66 1097415 336 336 query67 15839 15266 15289 15266 query68 7945864 508 508 query69 486 308 259 259 query70 1231113810721072 query71 415 292 262 262 query72 5565355637183556 query73 717 711 363 363 query74 9110941488698869 query75 3387319526952695 query76 33121175746 746 query77 770 415 283 283 query78 10080 10019 93629362 query79 2327840 598 598 query80 614 519 446 446 query81 501 257 229 229 query82 508 127 91 91 query83 173 171 156 156 query84 241 96 77 77 query85 800 367 314 314 query86 378 302 293 293 query87 4472447543024302 query88 4073225922582258 query89 396 320 288 288 query90 1939215 217 215 query91 144 139 110 110 query92 73 60 57 57 query93 18671072582 582 query94 707 444 298 298 query95 363 273 263 263 query96 494 568 277 277 query97 3334341833343334 query98 234 200 212 200 query99 1334142512781278 Total cold run time: 274237 ms Total hot run time: 185736 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2710005752 ClickBench: Total hot run time: 30.79 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 4005ab59daead76c5b48bc4319cb52d6edb9fbe1, data reload: false query1 0.030.030.04 query2 0.070.030.03 query3 0.230.060.06 query4 1.630.100.11 query5 0.590.560.55 query6 1.170.710.71 query7 0.020.020.02 query8 0.040.030.03 query9 0.590.530.52 query10 0.570.610.57 query11 0.160.110.11 query12 0.140.110.11 query13 0.640.610.60 query14 2.692.682.71 query15 0.930.860.85 query16 0.410.390.38 query17 1.031.041.05 query18 0.210.200.20 query19 2.011.981.80 query20 0.010.010.01 query21 15.35 0.930.56 query22 0.771.190.66 query23 14.94 1.390.62 query24 6.881.780.76 query25 0.480.090.18 query26 0.670.160.13 query27 0.050.060.05 query28 9.000.860.44 query29 12.57 4.023.31 query30 0.240.090.06 query31 2.830.590.40 query32 3.260.560.47 query33 2.972.983.04 query34 15.68 5.174.51 query35 4.574.554.59 query36 0.660.480.48 query37 0.080.060.06 query38 0.060.040.04 query39 0.030.020.03 query40 0.160.140.13 query41 0.080.020.02 query42 0.030.020.02 query43 0.040.040.03 Total cold run time: 104.57 s Total hot run time: 30.79 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2710401496 ClickBench: Total hot run time: 30.94 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 9895c431a8a55d9f0c9edf120705e65b2de79734, data reload: false query1 0.030.030.03 query2 0.070.040.04 query3 0.230.070.06 query4 1.620.100.11 query5 0.560.550.55 query6 1.180.710.71 query7 0.020.020.02 query8 0.040.040.03 query9 0.580.540.52 query10 0.570.610.58 query11 0.160.110.10 query12 0.140.110.11 query13 0.620.610.60 query14 2.702.672.69 query15 0.920.860.84 query16 0.390.380.38 query17 1.011.021.07 query18 0.220.200.19 query19 1.911.961.86 query20 0.020.010.01 query21 15.35 0.870.55 query22 0.781.130.69 query23 14.96 1.370.62 query24 7.760.891.07 query25 0.520.190.12 query26 0.680.160.13 query27 0.060.070.04 query28 9.480.820.43 query29 12.57 4.023.32 query30 0.250.090.07 query31 2.820.580.38 query32 3.250.550.46 query33 2.942.973.05 query34 15.82 5.144.56 query35 4.534.554.50 query36 0.660.490.48 query37 0.090.060.06 query38 0.050.030.03 query39 0.030.020.02 query40 0.180.140.13 query41 0.080.020.02 query42 0.040.020.02 query43 0.030.030.03 Total cold run time: 105.92 s Total hot run time: 30.94 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
Tech-Circle-48 commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2713249129 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2714900104 # BE UT Coverage Report Increment line coverage `0.00% (0/34)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/66537c74e9b00ed7930a1f7346b41ab40c9784b7_66537c74e9b00ed7930a1f7346b41ab40c9784b7/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/66537c74e9b00ed7930a1f7346b41ab40c9784b7_66537c74e9b00ed7930a1f7346b41ab40c9784b7/report/index.html) | Category | Coverage | |---|| | Function Coverage | 46.88% (12539/26745) | | Line Coverage | 36.60% (107175/292845) | | Region Coverage | 35.65% (54800/153696) | | Branch Coverage | 31.06% (27609/88898) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2714627863 TPC-H: Total hot run time: 32380 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 66537c74e9b00ed7930a1f7346b41ab40c9784b7, data reload: false -- Round 1 -- q1 17640 518650295029 q2 2052290 164 164 q3 10425 1289705 705 q4 10215 997 528 528 q5 7516236423222322 q6 199 165 135 135 q7 908 742 628 628 q8 9321135710221022 q9 4933475846994699 q10 6809231218901890 q11 470 274 257 257 q12 340 346 220 220 q13 1 367630943094 q14 251 230 217 217 q15 520 490 483 483 q16 631 598 608 598 q17 559 850 345 345 q18 6956650463786378 q19 1228965 560 560 q20 332 332 197 197 q21 2872222719431943 q22 10561050966 966 Total cold run time: 103010 ms Total hot run time: 32380 ms - Round 2, with runtime_filter_mode=off - q1 5122510951025102 q2 228 336 228 228 q3 2142264322832283 q4 1405181113921392 q5 4211415341544153 q6 208 167 127 127 q7 1860180316931693 q8 2510245625732456 q9 6777679267846784 q10 2864305626372637 q11 586 486 473 473 q12 630 742 578 578 q13 3265375531163116 q14 266 287 255 255 q15 501 468 460 460 q16 604 670 628 628 q17 1093158812901290 q18 7303709771947097 q19 758 764 816 764 q20 1895197917741774 q21 5112474545724572 q22 1097101010011001 Total cold run time: 50437 ms Total hot run time: 48863 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2714690711 ClickBench: Total hot run time: 30.73 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 66537c74e9b00ed7930a1f7346b41ab40c9784b7, data reload: false query1 0.040.030.03 query2 0.070.030.04 query3 0.240.060.06 query4 1.630.110.10 query5 0.560.560.58 query6 1.200.700.73 query7 0.020.020.02 query8 0.040.030.03 query9 0.570.510.52 query10 0.580.610.58 query11 0.160.110.11 query12 0.140.110.11 query13 0.630.610.60 query14 2.672.732.82 query15 0.920.850.84 query16 0.390.370.38 query17 1.021.041.00 query18 0.210.210.20 query19 1.851.761.98 query20 0.020.010.02 query21 15.36 0.860.54 query22 0.761.270.77 query23 14.77 1.400.63 query24 7.101.710.74 query25 0.480.200.10 query26 0.670.160.13 query27 0.050.050.06 query28 9.070.860.42 query29 12.54 4.023.33 query30 0.250.090.06 query31 2.820.570.38 query32 3.230.550.47 query33 3.082.973.06 query34 15.75 5.124.48 query35 4.524.534.51 query36 0.660.490.48 query37 0.080.060.06 query38 0.050.040.03 query39 0.030.030.03 query40 0.170.130.14 query41 0.080.020.03 query42 0.040.030.02 query43 0.040.030.03 Total cold run time: 104.56 s Total hot run time: 30.73 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2714674148 TPC-DS: Total hot run time: 185743 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 66537c74e9b00ed7930a1f7346b41ab40c9784b7, data reload: false query1 1005390 376 376 query2 6551191318781878 query3 6799220 211 211 query4 26249 23567 23288 23288 query5 5639695 510 510 query6 309 207 178 178 query7 4600495 294 294 query8 301 249 232 232 query9 8643263026442630 query10 542 313 257 257 query11 15512 15353 14885 14885 query12 165 109 108 108 query13 1682525 427 427 query14 10642 648063516351 query15 244 192 175 175 query16 7737647 454 454 query17 1566741 576 576 query18 2022410 314 314 query19 309 179 153 153 query20 118 113 122 113 query21 207 126 108 108 query22 4108442940714071 query23 33999 33148 33038 33038 query24 7534237624292376 query25 511 465 395 395 query26 1219272 155 155 query27 2586474 325 325 query28 4359243324282428 query29 710 545 418 418 query30 286 220 194 194 query31 943 848 800 800 query32 74 65 67 65 query33 554 361 350 350 query34 783 837 499 499 query35 788 833 754 754 query36 977 987 882 882 query37 118 99 75 75 query38 4253413940974097 query39 1431141914051405 query40 209 110 102 102 query41 54 51 56 51 query42 119 99 103 99 query43 495 522 482 482 query44 1278798 785 785 query45 178 174 168 168 query46 824 1018626 626 query47 1778178017541754 query48 387 414 303 303 query49 754 514 422 422 query50 695 733 409 409 query51 4168416340794079 query52 103 117 98 98 query53 245 254 192 192 query54 484 491 409 409 query55 82 81 81 81 query56 262 260 258 258 query57 1135116010671067 query58 246 263 229 229 query59 2476258325762576 query60 304 272 269 269 query61 126 121 134 121 query62 790 734 651 651 query63 242 185 186 185 query64 4292996 669 669 query65 4408429343174293 query66 989 413 290 290 query67 15796 15690 15500 15500 query68 8349879 515 515 query69 506 305 265 265 query70 1170113210511051 query71 499 290 265 265 query72 5512356137783561 query73 800 738 433 433 query74 9162892789188918 query75 3936317327152715 query76 37751186730 730 query77 787 357 284 284 query78 10201 10152 92669266 query79 2573814 584 584 query80 663 517 457 457 query81 497 255 218 218 query82 505 125 104 104 query83 168 256 149 149 query84 243 91 69 69 query85 744 347 314 314 query86 347 282 288 282 query87 4368440345214403 query88 3760229122732273 query89 391 322 281 281 query90 1951207 210 207 query91 138 145 110 110 query92 79 58 58 58 query93 21541064584 584 query94 670 401 311 311 query95 351 270 258 258 query96 480 568 276 276 query97 3307342832833283 query98 234 215 205 205 query99 1344137913011301 Total cold run time: 278658 ms Total hot run time: 185743 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
Tech-Circle-48 commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2714512581 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
Tech-Circle-48 commented on code in PR #48866:
URL: https://github.com/apache/doris/pull/48866#discussion_r1989415624
##
be/src/olap/delta_writer.cpp:
##
@@ -231,7 +239,26 @@ int64_t BaseDeltaWriter::mem_consumption(MemType mem) {
return _memtable_writer->mem_consumption(mem);
}
-void DeltaWriter::_request_slave_tablet_pull_rowset(const PNodeInfo&
node_info) {
+Status safe_get_file_size(const std::string& file_path, int64_t* file_size) {
Review Comment:
already simplify
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
Tech-Circle-48 commented on code in PR #48866:
URL: https://github.com/apache/doris/pull/48866#discussion_r1988766409
##
be/src/olap/delta_writer.cpp:
##
@@ -187,10 +187,18 @@ Status DeltaWriter::commit_txn(const PSlaveTabletNodes&
slave_tablet_nodes) {
SCOPED_TIMER(_commit_txn_timer);
RETURN_IF_ERROR(rowset_builder()->commit_txn());
+Status final_status = Status::OK();
for (auto&& node_info : slave_tablet_nodes.slave_nodes()) {
-_request_slave_tablet_pull_rowset(node_info);
+Status status = _request_slave_tablet_pull_rowset(node_info);
Review Comment:
thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2713709974 # BE UT Coverage Report Increment line coverage `4.65% (2/43)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/e3df906194bf863149d692c96033225dd00f7ee8_e3df906194bf863149d692c96033225dd00f7ee8/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/e3df906194bf863149d692c96033225dd00f7ee8_e3df906194bf863149d692c96033225dd00f7ee8/report/index.html) | Category | Coverage | |---|| | Function Coverage | 46.88% (12537/26745) | | Line Coverage | 36.60% (107171/292853) | | Region Coverage | 35.64% (54781/153699) | | Branch Coverage | 31.05% (27602/88902) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2713392511 TPC-DS: Total hot run time: 186255 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit e3df906194bf863149d692c96033225dd00f7ee8, data reload: false query1 998 419 384 384 query2 6532202819981998 query3 6791233 226 226 query4 26225 23658 23137 23137 query5 4451670 500 500 query6 297 205 191 191 query7 4611519 305 305 query8 303 242 246 242 query9 8669265026622650 query10 440 312 265 265 query11 15968 15144 14875 14875 query12 178 109 108 108 query13 1658540 420 420 query14 10820 663869226638 query15 213 199 178 178 query16 7715654 480 480 query17 1636782 584 584 query18 2004401 318 318 query19 221 195 166 166 query20 127 122 114 114 query21 212 140 107 107 query22 4146437440884088 query23 33913 32839 32842 32839 query24 7209240923852385 query25 519 503 387 387 query26 1219285 156 156 query27 2124489 331 331 query28 4008241623992399 query29 719 567 425 425 query30 287 226 193 193 query31 989 897 784 784 query32 73 69 64 64 query33 557 379 292 292 query34 803 859 520 520 query35 790 833 741 741 query36 957 995 902 902 query37 120 130 74 74 query38 4160428143464281 query39 1426141013941394 query40 211 117 104 104 query41 55 57 50 50 query42 118 104 109 104 query43 511 524 514 514 query44 1346802 793 793 query45 182 174 165 165 query46 889 1036644 644 query47 1737177117091709 query48 367 424 309 309 query49 776 509 421 421 query50 726 739 423 423 query51 4264421041644164 query52 109 107 101 101 query53 234 278 208 208 query54 495 506 417 417 query55 86 84 78 78 query56 273 283 266 266 query57 1099115710551055 query58 259 247 241 241 query59 2588284826602660 query60 297 287 256 256 query61 141 121 121 121 query62 801 711 670 670 query63 232 203 199 199 query64 42611020673 673 query65 4507435643984356 query66 1059421 296 296 query67 15753 15509 15086 15086 query68 8365892 522 522 query69 469 342 274 274 query70 1263113311251125 query71 462 301 276 276 query72 5651363336043604 query73 798 768 361 361 query74 9330913889908990 query75 3854317726932693 query76 36821192763 763 query77 767 400 282 282 query78 10027 10131 92949294 query79 3028842 586 586 query80 624 541 458 458 query81 488 264 222 222 query82 688 131 95 95 query83 173 176 165 165 query84 242 95 78 78 query85 766 364 312 312 query86 389 275 316 275 query87 4431450345214503 query88 3661231722802280 query89 397 331 291 291 query90 1861209 213 209 query91 151 138 110 110 query92 78 59 53 53 query93 17231105583 583 query94 668 397 318 318 query95 368 268 264 264 query96 490 564 283 283 query97 3326334832853285 query98 262 201 216 201 query99 1640138212761276 Total cold run time: 277586 ms Total hot run time: 186255 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2713416783 ClickBench: Total hot run time: 30.76 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit e3df906194bf863149d692c96033225dd00f7ee8, data reload: false query1 0.040.030.03 query2 0.070.040.03 query3 0.240.060.06 query4 1.620.100.10 query5 0.550.560.54 query6 1.160.720.72 query7 0.020.010.01 query8 0.040.030.04 query9 0.590.540.51 query10 0.570.640.58 query11 0.150.110.11 query12 0.150.120.11 query13 0.610.610.60 query14 2.682.702.73 query15 0.940.860.84 query16 0.390.360.39 query17 1.031.031.02 query18 0.210.200.20 query19 1.902.031.80 query20 0.020.010.01 query21 15.35 0.890.54 query22 0.751.120.59 query23 15.08 1.390.61 query24 6.672.410.72 query25 0.530.250.24 query26 0.580.150.13 query27 0.060.050.05 query28 9.870.830.42 query29 12.61 3.993.25 query30 0.260.090.06 query31 2.810.610.39 query32 3.230.560.46 query33 2.933.093.06 query34 15.65 5.174.54 query35 4.554.554.54 query36 0.670.490.48 query37 0.080.060.07 query38 0.060.040.04 query39 0.020.020.03 query40 0.160.130.13 query41 0.080.020.03 query42 0.040.020.02 query43 0.030.030.03 Total cold run time: 105.05 s Total hot run time: 30.76 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2713344407 TPC-H: Total hot run time: 32983 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit e3df906194bf863149d692c96033225dd00f7ee8, data reload: false -- Round 1 -- q1 17608 534251195119 q2 2049299 170 170 q3 10412 1389744 744 q4 10211 1033568 568 q5 7490245523712371 q6 192 160 132 132 q7 927 773 615 615 q8 9307130211581158 q9 4931477449034774 q10 6838231119111911 q11 493 285 283 283 q12 363 371 223 223 q13 17771 388031583158 q14 234 243 209 209 q15 541 484 485 484 q16 632 633 593 593 q17 588 890 352 352 q18 7222654963356335 q19 1089958 590 590 q20 353 353 210 210 q21 3027231619811981 q22 1026100510031003 Total cold run time: 103304 ms Total hot run time: 32983 ms - Round 2, with runtime_filter_mode=off - q1 5204513152105131 q2 250 327 221 221 q3 2199271723172317 q4 1534184114481448 q5 4237415841384138 q6 208 160 125 125 q7 2028196317781778 q8 2660259925882588 q9 7254710971837109 q10 2962323827442744 q11 566 512 486 486 q12 696 821 650 650 q13 3576393331823182 q14 285 286 275 275 q15 515 483 465 465 q16 652 692 656 656 q17 1159159513651365 q18 7914749573857385 q19 872 853 896 853 q20 1986199818861886 q21 5625494347784778 q22 10901053991 991 Total cold run time: 53472 ms Total hot run time: 50571 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
sollhui commented on code in PR #48866:
URL: https://github.com/apache/doris/pull/48866#discussion_r1988761510
##
be/src/olap/delta_writer.cpp:
##
@@ -187,10 +187,18 @@ Status DeltaWriter::commit_txn(const PSlaveTabletNodes&
slave_tablet_nodes) {
SCOPED_TIMER(_commit_txn_timer);
RETURN_IF_ERROR(rowset_builder()->commit_txn());
+Status final_status = Status::OK();
for (auto&& node_info : slave_tablet_nodes.slave_nodes()) {
-_request_slave_tablet_pull_rowset(node_info);
+Status status = _request_slave_tablet_pull_rowset(node_info);
Review Comment:
RETURN_IF_ERROR(_request_slave_tablet_pull_rowset(node_info))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
Tech-Circle-48 commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2710308946 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
Tech-Circle-48 commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2710251885 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method [doris]
doris-robot commented on PR #48866: URL: https://github.com/apache/doris/pull/48866#issuecomment-2709956131 TPC-H: Total hot run time: 32580 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 4005ab59daead76c5b48bc4319cb52d6edb9fbe1, data reload: false -- Round 1 -- q1 17600 525650995099 q2 2051308 166 166 q3 10408 1352709 709 q4 10220 1008557 557 q5 7521244223502350 q6 190 163 130 130 q7 926 722 603 603 q8 9294122011251125 q9 5132483246564656 q10 6844231718961896 q11 465 264 253 253 q12 360 360 221 221 q13 17783 369232033203 q14 234 235 220 220 q15 530 475 482 475 q16 620 626 580 580 q17 578 872 351 351 q18 6932641662666266 q19 1234954 556 556 q20 318 333 202 202 q21 2842222819841984 q22 10221012978 978 Total cold run time: 103104 ms Total hot run time: 32580 ms - Round 2, with runtime_filter_mode=off - q1 5185513351085108 q2 243 331 233 233 q3 2151271123382338 q4 1512184613881388 q5 4241411542184115 q6 210 163 123 123 q7 1848198417481748 q8 2628254525302530 q9 7334720470937093 q10 3001322427892789 q11 594 513 479 479 q12 694 782 631 631 q13 3594392032583258 q14 270 296 275 275 q15 512 476 464 464 q16 657 683 642 642 q17 1163155513871387 q18 7731764673147314 q19 821 835 920 835 q20 1989202318661866 q21 5461508049014901 q22 1133104810101010 Total cold run time: 52972 ms Total hot run time: 50527 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
