Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
gavinchou merged PR #52677: URL: https://github.com/apache/doris/pull/52677 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3241010372 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
hello-stephen commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3241045947 # Cloud UT Coverage Report Increment line coverage `100.00% (37/37)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/29a57453e6b024f67604e973c93f743aefbd2b30_29a57453e6b024f67604e973c93f743aefbd2b30_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/29a57453e6b024f67604e973c93f743aefbd2b30_29a57453e6b024f67604e973c93f743aefbd2b30_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 84.49% (1465/1734) | | Line Coverage | 67.69% (26155/38640) | | Region Coverage | 68.62% (12985/18922) | | Branch Coverage | 58.58% (6945/11856) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3240839851 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
github-actions[bot] commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3182145829 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
github-actions[bot] commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3182145763 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
doris-robot commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3178095798 # Cloud UT Coverage Report Increment line coverage `100.00% (28/28)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/bcf950223c821756ebb3afede4d2544387b02106_bcf950223c821756ebb3afede4d2544387b02106_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/bcf950223c821756ebb3afede4d2544387b02106_bcf950223c821756ebb3afede4d2544387b02106_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 83.61% (1423/1702) | | Line Coverage | 66.84% (24236/36261) | | Region Coverage | 67.86% (12066/17780) | | Branch Coverage | 57.47% (6337/11026) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3178003869 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3177462069 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
doris-robot commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3094603691 # Cloud UT Coverage Report Increment line coverage `100.00% (22/22)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/d48ef4207e395dca14fb5014c0b2e9b4e065157a_d48ef4207e395dca14fb5014c0b2e9b4e065157a_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/d48ef4207e395dca14fb5014c0b2e9b4e065157a_d48ef4207e395dca14fb5014c0b2e9b4e065157a_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 80.37% (1298/1615) | | Line Coverage | 65.79% (21748/33058) | | Region Coverage | 67.11% (10924/16277) | | Branch Coverage | 56.69% (5749/10142) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3094592852 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
gavinchou commented on code in PR #52677:
URL: https://github.com/apache/doris/pull/52677#discussion_r2199780642
##
cloud/src/recycler/recycler.cpp:
##
@@ -2771,28 +2779,36 @@ int InstanceRecycler::recycle_tmp_rowsets() {
return 0;
};
-auto loop_done = [&tmp_rowset_keys, &tmp_rowsets, &num_recycled,
&metrics_context,
- this]() -> int {
+auto loop_done = [&, this]() -> int {
DORIS_CLOUD_DEFER {
tmp_rowset_keys.clear();
tmp_rowsets.clear();
};
-if (delete_rowset_data(tmp_rowsets, RowsetRecyclingState::TMP_ROWSET,
metrics_context) !=
-0) {
-LOG(WARNING) << "failed to delete tmp rowset data, instance_id="
<< instance_id_;
-return -1;
-}
-if (txn_remove(txn_kv_.get(), tmp_rowset_keys) != 0) {
-LOG(WARNING) << "failed to delete tmp rowset kv, instance_id=" <<
instance_id_;
-return -1;
-}
-num_recycled += tmp_rowset_keys.size();
+worker_pool->submit([&, tmp_rowset_keys_to_delete = tmp_rowset_keys,
+ tmp_rowsets_to_delete = tmp_rowsets]() {
+if (delete_rowset_data(tmp_rowsets_to_delete,
RowsetRecyclingState::TMP_ROWSET,
+ metrics_context) != 0) {
+LOG(WARNING) << "failed to delete tmp rowset data,
instance_id=" << instance_id_;
+return;
+}
+if (txn_remove(txn_kv_.get(), tmp_rowset_keys_to_delete) != 0) {
+LOG(WARNING) << "failed to delete tmp rowset kv, instance_id="
<< instance_id_;
+return;
+}
+LOG(INFO) << "finish recycle tmp rowsets, num_recycled="
Review Comment:
too many logs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
gavinchou commented on code in PR #52677:
URL: https://github.com/apache/doris/pull/52677#discussion_r2197006900
##
cloud/src/recycler/recycler.cpp:
##
@@ -2654,10 +2659,13 @@ int InstanceRecycler::recycle_tmp_rowsets() {
};
// Elements in `tmp_rowset_keys` has the same lifetime as `it`
-std::vector tmp_rowset_keys;
+std::vector tmp_rowset_keys;
// rowset_id -> rowset_meta
// store tmp_rowset id and meta for statistics rs size when delete
std::map tmp_rowsets;
+auto worker_pool = std::make_unique(
+config::instance_recycler_worker_pool_size, "recycle_tmp_rowsets");
Review Comment:
why not use SyncExecutor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
doris-robot commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3035046619 # Cloud UT Coverage Report Increment line coverage `100.00% (29/29)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/79f4f62b2fa39f014a60bcf6ec5034a2826f7fa8_79f4f62b2fa39f014a60bcf6ec5034a2826f7fa8_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/79f4f62b2fa39f014a60bcf6ec5034a2826f7fa8_79f4f62b2fa39f014a60bcf6ec5034a2826f7fa8_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 82.93% (1219/1470) | | Line Coverage | 67.51% (21062/31200) | | Region Coverage | 67.26% (10485/15589) | | Branch Coverage | 56.64% (5494/9700) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3035010767 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3034922315 run cloudut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3034837767 run cloudut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
doris-robot commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3034823923 # Cloud UT Coverage Report Increment line coverage `100.00% (32/32)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/c5b2f1bb0dc1dd4228ed38b1045f153614763680_c5b2f1bb0dc1dd4228ed38b1045f153614763680_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/c5b2f1bb0dc1dd4228ed38b1045f153614763680_c5b2f1bb0dc1dd4228ed38b1045f153614763680_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 82.93% (1219/1470) | | Line Coverage | 67.58% (21088/31203) | | Region Coverage | 67.27% (10486/15589) | | Branch Coverage | 56.60% (5490/9700) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3034775636 run cloudut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3034777000 run cloudut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3034735559 run cloudut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3034314563 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3034261584 run cloudut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3034240341 run cloud_ut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3031674839 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
Thearas commented on PR #52677: URL: https://github.com/apache/doris/pull/52677#issuecomment-3027260062 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
wyxxxcat closed pull request #52307: [opt](recycler) Add concurrency recycle for tmp rowset URL: https://github.com/apache/doris/pull/52307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Add concurrency recycle for tmp rowset [doris]
hello-stephen commented on PR #52307: URL: https://github.com/apache/doris/pull/52307#issuecomment-3004312818 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
