Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
gavinchou commented on PR #52816: URL: https://github.com/apache/doris/pull/52816#issuecomment-3038334702 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
gavinchou commented on code in PR #52816:
URL: https://github.com/apache/doris/pull/52816#discussion_r2186893899
##
be/src/cloud/cloud_meta_mgr.cpp:
##
@@ -131,6 +131,22 @@ Status bthread_fork_join(const
std::vector>& tasks, int
return status;
}
+void bthread_fork_join(const std::vector>& tasks, int
concurrency,
+ std::future* fut) {
+// std::function will cause `copy`, we need to use heap memory to avoid
copy ctor called
+auto prom = std::make_shared>();
+*fut = prom->get_future();
+std::function* fn =
+new std::function([&tasks, concurrency, p =
std::move(prom)]() mutable {
+p->set_value(bthread_fork_join(tasks, concurrency));
+});
+
+bthread_t bthread_id;
+if (bthread_start_background(&bthread_id, nullptr, run_bthread_work, fn)
!= 0) {
+run_bthread_work(fn);
Review Comment:
也是可以的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
gavinchou commented on code in PR #52816: URL: https://github.com/apache/doris/pull/52816#discussion_r2186893768 ## be/src/cloud/cloud_meta_mgr.cpp: ## @@ -131,6 +131,22 @@ Status bthread_fork_join(const std::vector>& tasks, int return status; } +void bthread_fork_join(const std::vector>& tasks, int concurrency, Review Comment: caller 使用返回的fut去调用 future.wait_for(tmout) 就可以了 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
yiguolei commented on code in PR #52816: URL: https://github.com/apache/doris/pull/52816#discussion_r2186284083 ## be/src/cloud/cloud_meta_mgr.cpp: ## @@ -131,6 +131,22 @@ Status bthread_fork_join(const std::vector>& tasks, int return status; } +void bthread_fork_join(const std::vector>& tasks, int concurrency, Review Comment: 这个函数得有一个timeout 参数，如果超过一定时间，得失败 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
yiguolei commented on code in PR #52816:
URL: https://github.com/apache/doris/pull/52816#discussion_r2186283809
##
be/src/cloud/cloud_meta_mgr.cpp:
##
@@ -131,6 +131,22 @@ Status bthread_fork_join(const
std::vector>& tasks, int
return status;
}
+void bthread_fork_join(const std::vector>& tasks, int
concurrency,
+ std::future* fut) {
+// std::function will cause `copy`, we need to use heap memory to avoid
copy ctor called
+auto prom = std::make_shared>();
+*fut = prom->get_future();
+std::function* fn =
+new std::function([&tasks, concurrency, p =
std::move(prom)]() mutable {
+p->set_value(bthread_fork_join(tasks, concurrency));
+});
+
+bthread_t bthread_id;
+if (bthread_start_background(&bthread_id, nullptr, run_bthread_work, fn)
!= 0) {
+run_bthread_work(fn);
Review Comment:
这里如果145 行启动bthread 失败，我们这里这里直接返回 error status？没必要再尝试一次了吧？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
hello-stephen commented on PR #52816: URL: https://github.com/apache/doris/pull/52816#issuecomment-3037055432 # BE UT Coverage Report Increment line coverage `88.89% (16/18)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/e7c70150bdb616ce81c58330ba676cc9412c64fc_e7c70150bdb616ce81c58330ba676cc9412c64fc/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/e7c70150bdb616ce81c58330ba676cc9412c64fc_e7c70150bdb616ce81c58330ba676cc9412c64fc/report/index.html) | Category | Coverage | |---|| | Function Coverage | 57.34% (15487/27008) | | Line Coverage | 46.32% (140708/303780) | | Region Coverage | 45.60% (71184/156116) | | Branch Coverage | 40.31% (37504/93038) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
doris-robot commented on PR #52816: URL: https://github.com/apache/doris/pull/52816#issuecomment-3036949312 ClickBench: Total hot run time: 28.95 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit e7c70150bdb616ce81c58330ba676cc9412c64fc, data reload: false query1 0.050.040.03 query2 0.070.050.04 query3 0.250.080.09 query4 1.600.110.11 query5 0.450.430.42 query6 1.160.650.66 query7 0.020.020.02 query8 0.040.040.04 query9 0.590.510.52 query10 0.580.570.57 query11 0.170.110.11 query12 0.140.120.11 query13 0.630.630.61 query14 0.790.800.80 query15 0.890.860.88 query16 0.400.380.40 query17 1.041.081.05 query18 0.220.220.20 query19 1.951.811.93 query20 0.020.010.01 query21 15.39 0.880.56 query22 0.771.210.65 query23 14.97 1.380.61 query24 7.071.250.32 query25 0.420.190.08 query26 0.580.160.15 query27 0.060.060.05 query28 9.180.920.44 query29 12.53 3.953.30 query30 0.240.090.06 query31 2.850.590.39 query32 3.240.570.47 query33 3.143.083.14 query34 16.06 5.464.73 query35 4.844.854.82 query36 0.660.510.49 query37 0.100.060.07 query38 0.060.040.04 query39 0.030.020.02 query40 0.180.150.14 query41 0.080.020.02 query42 0.030.030.03 query43 0.040.030.03 Total cold run time: 103.58 s Total hot run time: 28.95 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
doris-robot commented on PR #52816: URL: https://github.com/apache/doris/pull/52816#issuecomment-3036941911 TPC-DS: Total hot run time: 184461 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit e7c70150bdb616ce81c58330ba676cc9412c64fc, data reload: false query1 1005383 396 383 query2 6514168016391639 query3 6736210 209 209 query4 26392 23549 23380 23380 query5 4349573 423 423 query6 302 206 215 206 query7 4618491 289 289 query8 285 225 218 218 query9 8624259425882588 query10 463 317 283 283 query11 15226 15100 14795 14795 query12 157 109 103 103 query13 1654523 405 405 query14 8546559456735594 query15 198 183 188 183 query16 7373603 462 462 query17 1204726 589 589 query18 1999410 304 304 query19 209 188 173 173 query20 123 113 111 111 query21 214 120 106 106 query22 4409448541884188 query23 33736 33015 33062 33015 query24 8550234623442344 query25 538 460 407 407 query26 1219262 143 143 query27 2764514 327 327 query28 4364214421012101 query29 756 557 432 432 query30 280 212 200 200 query31 913 839 749 749 query32 70 64 59 59 query33 557 366 297 297 query34 787 829 522 522 query35 769 797 732 732 query36 919 982 889 889 query37 119 110 82 82 query38 4100402840274027 query39 1468141113931393 query40 214 114 102 102 query41 54 56 55 55 query42 125 122 101 101 query43 477 500 493 493 query44 1300831 842 831 query45 176 171 166 166 query46 838 1026623 623 query47 1782178717321732 query48 383 413 296 296 query49 746 480 392 392 query50 626 680 412 412 query51 4152407240394039 query52 115 106 97 97 query53 216 246 187 187 query54 561 553 493 493 query55 80 76 110 76 query56 290 296 270 270 query57 1168119111311131 query58 265 242 257 242 query59 2634262925092509 query60 327 315 297 297 query61 123 121 121 121 query62 832 714 645 645 query63 222 182 186 182 query64 42861003654 654 query65 4282416441744164 query66 1140406 306 306 query67 15803 15610 15431 15431 query68 8299886 524 524 query69 467 299 261 261 query70 1269105710811057 query71 495 315 309 309 query72 5579474447214721 query73 700 616 352 352 query74 9311924188448844 query75 3834318627532753 query76 36381129687 687 query77 779 384 272 272 query78 10144 10077 92509250 query79 5122806 567 567 query80 691 499 452 452 query81 462 246 219 219 query82 718 123 95 95 query83 283 240 234 234 query84 294 109 79 79 query85 781 355 313 313 query86 322 376 267 267 query87 4482432142864286 query88 2879227723032277 query89 446 318 278 278 query90 1925203 199 199 query91 139 137 109 109 query92 78 60 55 55 query93 2759932 577 577 query94 672 469 295 295 query95 366 289 279 279 query96 486 560 278 278 query97 2751277026762676 query98 230 205 206 205 query99 1464138612941294 Total cold run time: 277591 ms Total hot run time: 184461 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
doris-robot commented on PR #52816: URL: https://github.com/apache/doris/pull/52816#issuecomment-3036922781 TPC-H: Total hot run time: 33513 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit e7c70150bdb616ce81c58330ba676cc9412c64fc, data reload: false -- Round 1 -- q1 17567 520350335033 q2 1915280 176 176 q3 10336 1276681 681 q4 10214 1005505 505 q5 7507241223002300 q6 176 159 127 127 q7 878 734 588 588 q8 9300122610041004 q9 7155509551775095 q10 6858239519751975 q11 460 274 274 274 q12 342 339 210 210 q13 17765 366730453045 q14 218 222 207 207 q15 540 474 486 474 q16 419 422 380 380 q17 586 820 370 370 q18 7509712270177017 q19 1205953 548 548 q20 336 349 222 222 q21 3632314223312331 q22 10251015951 951 Total cold run time: 105943 ms Total hot run time: 33513 ms - Round 2, with runtime_filter_mode=off - q1 5054510150705070 q2 235 321 221 221 q3 2143266522962296 q4 1368174113411341 q5 4170418746414187 q6 213 166 130 130 q7 2014195318481848 q8 2688265125362536 q9 7446727873387278 q10 3081327528622862 q11 608 500 508 500 q12 682 739 582 582 q13 3533393333133313 q14 314 308 278 278 q15 520 487 504 487 q16 445 498 466 466 q17 1170160013541354 q18 8015747676607476 q19 807 790 962 790 q20 2169204219021902 q21 4760442242444244 q22 1077102210001000 Total cold run time: 52512 ms Total hot run time: 50161 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
gavinchou commented on PR #52816: URL: https://github.com/apache/doris/pull/52816#issuecomment-3036867628 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
gavinchou commented on PR #52816: URL: https://github.com/apache/doris/pull/52816#issuecomment-3036865733 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](cloud) Add async wrap of `bthread_fork_join` with promise-future [doris]
Thearas commented on PR #52816: URL: https://github.com/apache/doris/pull/52816#issuecomment-3036864770 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
